Oct 30 23:53:08.189132 ip-10-0-143-16 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:53:08.624485 ip-10-0-143-16 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:53:08.624485 ip-10-0-143-16 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:53:08.624485 ip-10-0-143-16 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:53:08.624485 ip-10-0-143-16 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:53:08.629036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.628956    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.629036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629035    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.629036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629040    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629043    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629046    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629049    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629052    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629055    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629058    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629061    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629063    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629066    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629074    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629077    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629079    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629082    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629084    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629087    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629090    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629092    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629095    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629097    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.629125 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629100    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629102    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629105    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629107    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629110    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629112    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629115    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629118    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629120    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629122    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629127    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629131    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629135    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629138    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629142    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629145    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629147    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629151    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629153    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.630320 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629157    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629159    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629162    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629165    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629167    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629170    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629172    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629175    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629177    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629180    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629182    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629185    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629187    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629190    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629192    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629195    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629198    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629200    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629203    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629205    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.630765 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629208    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629210    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629213    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629216    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629219    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629222    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629224    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629226    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629229    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629231    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629234    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629236    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629239    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629241    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629243    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629246    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629248    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629251    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629253    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.631243 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629257    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629266    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629270    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629273    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629277    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629280    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629283    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629285    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629289    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629292    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629294    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629297    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629299    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629302    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629304    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629307    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629310    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629313    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629316    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629318    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.631698 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629321    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629323    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629326    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629328    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629331    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.629333    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630140    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630148    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630153    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630156    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630159    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630161    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630164    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630167    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630170    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630172    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630175    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630177    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630180    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630182    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.632182 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630185    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630188    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630191    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630193    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630196    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630198    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630201    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630203    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630207    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630210    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630213    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630216    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630218    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630222    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630224    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630226    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630229    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630231    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630234    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630236    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.632654 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630239    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630241    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630244    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630246    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630249    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630251    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630254    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630256    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630259    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630261    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630263    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630266    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630269    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630271    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630274    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630276    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630279    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630281    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630285    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630288    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.633149 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630290    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630292    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630295    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630298    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630301    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630303    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630306    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630309    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630311    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630314    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630316    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630319    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630321    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630324    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630326    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630329    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630332    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630334    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630337    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630339    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.633619 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630342    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630344    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630347    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630349    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630352    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630355    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630357    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630360    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630362    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630364    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630367    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630370    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630372    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630375    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630377    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630381    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630383    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630385    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630388    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.634114 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630392    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630395    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630398    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630401    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630406    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630409    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630411    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630414    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630416    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630418    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630421    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630423    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630426    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630903    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630909    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630912    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630915    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630918    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630921    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.634798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630924    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630926    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630929    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630931    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630934    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630936    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630939    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630942    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630944    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630952    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630956    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630958    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630962    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630964    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630967    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630970    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630972    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630977    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630979    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.635382 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630981    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630984    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630987    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630989    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630992    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630994    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630997    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.630999    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631002    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631004    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631008    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631011    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631014    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631017    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631020    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631022    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631025    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631027    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631030    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.636146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631032    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631035    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631037    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631040    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631043    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631045    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631048    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631050    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631053    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631055    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631057    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631064    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631066    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631069    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631072    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631075    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631077    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631080    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631083    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631085    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.636696 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631087    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631090    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631092    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631095    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631097    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631100    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631102    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631104    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631107    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631109    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631112    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631114    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631116    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631119    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631121    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631124    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631127    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631129    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631132    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631134    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.637353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631137    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631140    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631142    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631146    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631148    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631151    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631154    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631157    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631159    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631162    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631164    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631166    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631169    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631172    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631174    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631176    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631179    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631181    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631184    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.637999 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631186    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631188    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.631191    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633125    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633136    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633143    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633148    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633152    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633156    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633160    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633164    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633168    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633171    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633175    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633178    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633181    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633185    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633188    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633191    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633194    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633197    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633200    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633203    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633206    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:53:08.638480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633209    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633212    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633216    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633219    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633223    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633226    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633228    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633231    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633234    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633237    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633240    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633244    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633247    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633255    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633258    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633261    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633264    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633268    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633271    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633274    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633277    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633280    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633285    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633288    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633291    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:53:08.639155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633294    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633297    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633300    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633304    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633306    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633309    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633312    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633315    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633318    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633321    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633324    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633328    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633331    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633334    2578 flags.go:64] FLAG: --help="false"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633336    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633340    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633343    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633346    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633349    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633352    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633356    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633359    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633362    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:53:08.639779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633365    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633368    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633371    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633374    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633377    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633380    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633383    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633386    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633389    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633391    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633394    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633397    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633402    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633405    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633408    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633411    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633413    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633417    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633419    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633422    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633426    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633429    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633433    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633436    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633439    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:53:08.640369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633441    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633444    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633447    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633450    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633454    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633461    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633464    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633467    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633470    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633473    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633476    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633479    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633481    2578 flags.go:64] FLAG: --port="10250"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633485    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633488    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0111924f3c1f82f3f"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633492    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633495    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633497    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633500    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633504    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633507    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633509    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633512    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633516    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633519    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633521    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:53:08.640978 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633524    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633527    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633529    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633532    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633535    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633538    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633540    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633543    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633546    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633549    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633553    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633556    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633559    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633562    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633565    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633567    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633570    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633577    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633580    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633583    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633586    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633589    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633592    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633595    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633598    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:53:08.641589 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633600    2578 flags.go:64] FLAG: --v="2"
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633605    2578 flags.go:64] FLAG: --version="false"
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633609    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633613    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.633616    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633750    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633754    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633757    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633759    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633762    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633765    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633768    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633770    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633772    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633775    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633778    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633780    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633784    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633786    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633789    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633791    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.642328 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633794    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633796    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633799    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633816    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633819    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633822    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633824    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633827    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633830    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633832    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633835    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633838    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633840    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633843    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633845    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633848    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633850    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633852    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633855    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633857    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.642856 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633860    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633862    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633865    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633867    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633870    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633872    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633875    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633877    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633881    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633884    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633886    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633889    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633891    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633894    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633896    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633900    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633902    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633905    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633907    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.643338 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633910    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633913    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633917    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633921    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633925    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633928    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633931    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633934    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633937    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633939    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633942    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633944    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633947    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633949    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633952    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633954    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633957    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633959    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.643798 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633962    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633964    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633967    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633971    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633973    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633976    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633979    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633981    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633983    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633986    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633990    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633993    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633995    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.633997    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.634000    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.634003    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.634042    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.634079    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.634083    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635143    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.644254 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635167    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635174    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635179    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635184    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635189    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635194    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635198    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635202    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635207    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635211    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635216    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635220    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635230    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.635293    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635442    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635449    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.644737 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635454    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635458    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635462    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635471    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635476    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635480    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635485    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635488    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635491    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635494    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635497    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635499    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635502    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635505    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635508    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635510    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635517    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635520    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635524    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.645140 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635527    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635530    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635532    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635535    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635538    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635586    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635723    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635731    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635735    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635740    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635746    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635750    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635754    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635759    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635763    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635767    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635776    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635783    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.645671 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635790    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635796    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635818    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635823    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635829    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635833    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635837    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635842    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635846    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635850    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635854    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635858    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635863    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635868    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635873    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635881    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635885    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635889    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635894    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.646234 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635898    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635903    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635908    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635913    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635917    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635921    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635925    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635930    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635934    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635939    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635943    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635947    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635951    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635956    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635960    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635964    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635969    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635973    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635977    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635981    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.646682 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635985    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635990    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635994    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.635998    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636003    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636007    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636012    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636016    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636025    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636029    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636033    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636038    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636042    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636047    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636052    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636056    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636061    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636065    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636072    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636077    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.647170 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636081    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636086    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636090    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636094    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636098    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636102    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636106    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.636110    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.643861    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.643876    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.644643    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.644650    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644708    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644713    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644717    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644720    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644723    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644726    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644729    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644732    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.647687 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644735    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644737    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644740    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644743    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644746    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644749    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644752    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644754    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644757    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644760    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644762    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644765    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644767    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644770    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644772    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644775    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644778    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644781    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644784    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644787    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.648179 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644790    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644793    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644795    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644798    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644800    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644821    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644826    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644830    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644835    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644839    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644842    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644846    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644848    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644851    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644854    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644857    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644859    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644861    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644864    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644866    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.648649 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644869    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644871    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644874    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644876    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644879    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644881    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644884    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644886    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644889    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644892    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644895    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644897    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644900    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644903    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644905    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644908    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644911    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644913    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644916    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.649146 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644918    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644921    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644923    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644926    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644928    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644931    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644933    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644937    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644941    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644944    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644947    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644949    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644953    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644956    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644959    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644961    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644963    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644966    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644969    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.649597 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644971    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644974    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644977    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644980    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644983    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644985    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644988    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644991    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644993    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644996    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.644998    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645001    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645004    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645006    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645009    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645012    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645014    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645016    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645019    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.650045 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645021    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.645050    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645191    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645197    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645200    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645203    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645206    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645209    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645212    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645215    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645217    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645220    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645222    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645226    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645230    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645233    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.650499 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645235    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645238    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645241    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645244    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645246    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645249    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645251    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645254    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645256    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645258    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645261    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645265    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645268    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645272    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645274    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645278    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645280    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645283    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645286    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.650895 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645289    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645291    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645294    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645296    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645298    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645301    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645304    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645306    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645309    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645311    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645313    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645316    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645318    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645320    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645323    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645325    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645328    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645331    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645334    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645336    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.651353 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645339    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645341    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645344    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645346    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645349    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645351    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645354    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645356    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645359    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645361    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645363    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645366    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645368    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645371    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645373    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645375    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645378    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645380    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645383    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645386    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.651840 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645388    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645391    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645393    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645396    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645398    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645400    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645403    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645406    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645408    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645412    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645414    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645417    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645419    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645422    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645424    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645427    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645429    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645432    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645434    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.652312 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645436    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645439    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645441    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645444    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645446    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645449    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645452    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645454    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645456    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645459    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645461    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645464    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645467    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645469    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.645491    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645521    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:08.652758 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645525    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645528    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645530    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645533    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645536    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645538    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645541    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645544    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645546    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645549    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645551    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645554    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645557    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645559    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645562    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645564    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645566    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645569    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645572    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645574    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:08.653147 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645577    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645579    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645582    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645584    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645586    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645589    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645591    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645594    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645596    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645599    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645602    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645605    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645607    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645610    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645612    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645614    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645617    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645619    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645622    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645624    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:08.653614 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645628    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645632    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645635    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645638    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645640    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645643    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645645    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645648    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645650    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645653    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645655    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645657    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645660    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645662    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645666    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645669    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645673    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645676    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645679    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:08.654116 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645681    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645684    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645687    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645690    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645692    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645695    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645697    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645699    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645702    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645704    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645707    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645710    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645712    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645714    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645717    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645719    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645721    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645723    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645726    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645728    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:08.654554 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645731    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645733    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645736    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645738    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645740    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645743    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645745    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645747    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645750    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645752    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645755    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645757    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645759    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645761    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645764    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645767    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645770    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645773    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645775    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:08.655036 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645778    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645780    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645783    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645822    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645827    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645830    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:08.645833    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.645892    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.648377    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.649625    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.649719    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.649757    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.652092    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:53:08.655957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.655583    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:53:08.668259 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.668243    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:53:08.671406 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.671391    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 23:53:08.672202 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.672177    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:53:08.678581 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.678562    2578 fs.go:135] Filesystem UUIDs: map[4611be2a-145d-40c5-b010-c0f6aa0e0aa4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f1a614a9-ed6c-476c-a5bb-0a9c4b1703a5:/dev/nvme0n1p3]
Oct 30 23:53:08.678627 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.678580    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 23:53:08.684088 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.683968    2578 manager.go:217] Machine: {Timestamp:2026-10-30 23:53:08.683451217 +0000 UTC m=+0.377916070 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100685 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2458e279084deb2fe6e604e98b807e SystemUUID:ec2458e2-7908-4deb-2fe6-e604e98b807e BootID:6be6fa3e-79bf-4613-9e53-29d16925bb1f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:38:1f:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:38:1f:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:40:e7:65:e4:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:53:08.684730 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.684719    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:53:08.684857 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.684844    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:53:08.688420 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.688396    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:53:08.688548 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.688424    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-16.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:53:08.688596 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.688551    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:53:08.688596 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.688559    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:53:08.688596 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.688581    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:53:08.688668 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.688597    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:53:08.689852 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.689842    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:53:08.694543 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694530    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:53:08.694665 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694655    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:53:08.694700 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694689    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:53:08.694727 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694703    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:53:08.694727 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694722    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:53:08.694777 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694735    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:53:08.694777 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694746    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:53:08.695007 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.694989    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:53:08.695151 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.695136    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:53:08.697713 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.697694    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:53:08.705243 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.705227    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:53:08.706069 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706057    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:53:08.706150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706088    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:53:08.706945 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706933    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:53:08.706979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706949    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:53:08.706979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706956    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:53:08.706979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706962    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:53:08.706979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706967    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:53:08.706979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.706973    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:53:08.707682 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.707673    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:53:08.707682 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.707683    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:53:08.707734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.707690    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:53:08.707734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.707703    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:53:08.707734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.707715    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:53:08.707734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.707724    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:53:08.708559 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.708550    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:53:08.708559 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.708559    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:53:08.712594 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.712580    2578 server.go:1263] "Started kubelet"
Oct 30 23:53:08.712695 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.712651    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:53:08.713082 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.713030    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:53:08.713171 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.713098    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:53:08.713314 ip-10-0-143-16 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:53:08.713966 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.713935    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-16.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 23:53:08.714606 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.714564    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:53:08.714993 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.714907    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:53:08.720036 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.720013    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 23:53:08.723111 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.723085    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 23:53:08.723111 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.723090    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-16.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 23:53:08.723285 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.723260    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:53:08.723499 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.722370    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-16.us-east-2.compute.internal.18e373ca27d685b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-16.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-16.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:53:08.712547768 +0000 UTC m=+0.407012620,LastTimestamp:2026-10-30 23:53:08.712547768 +0000 UTC m=+0.407012620,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-16.us-east-2.compute.internal,}"
Oct 30 23:53:08.723869 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.723855    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:53:08.724050 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.723980    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:08.724643 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724624    2578 factory.go:55] Registering systemd factory
Oct 30 23:53:08.724734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724647    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:53:08.724734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724711    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:53:08.724734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724726    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:53:08.724734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724711    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:53:08.724918 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724817    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:53:08.724918 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724825    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:53:08.724918 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724883    2578 factory.go:153] Registering CRI-O factory
Oct 30 23:53:08.724918 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724894    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:53:08.725092 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724935    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:53:08.725092 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724954    2578 factory.go:103] Registering Raw factory
Oct 30 23:53:08.725092 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.724967    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:53:08.726042 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.726020    2578 manager.go:319] Starting recovery of all containers
Oct 30 23:53:08.737095 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.737061    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-16.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 23:53:08.739154 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.739136    2578 manager.go:324] Recovery completed
Oct 30 23:53:08.742775 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.742758    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 23:53:08.743352 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.743339    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.745125 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745103    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.745180 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745138    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.745180 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745149    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.745618 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745600    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:53:08.745618 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745616    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:53:08.745723 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745608    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.745723 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.745637    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:53:08.746311 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.746297    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.746360 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.746323    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.746360 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.746345    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.747950 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.747939    2578 policy_none.go:50] "Start"
Oct 30 23:53:08.747988 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.747954    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:53:08.747988 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.747967    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:53:08.749993 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.749978    2578 policy_none.go:44] "Start"
Oct 30 23:53:08.752613 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.752589    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ktxkb"
Oct 30 23:53:08.755639 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.755554    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-16.us-east-2.compute.internal.18e373ca29c79aa0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-16.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-16.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-16.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:53:08.745124512 +0000 UTC m=+0.439589365,LastTimestamp:2026-10-30 23:53:08.745124512 +0000 UTC m=+0.439589365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-16.us-east-2.compute.internal,}"
Oct 30 23:53:08.760681 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.760604    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-16.us-east-2.compute.internal.18e373ca29c7e517  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-16.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-16.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-16.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:53:08.745143575 +0000 UTC m=+0.439608427,LastTimestamp:2026-10-30 23:53:08.745143575 +0000 UTC m=+0.439608427,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-16.us-east-2.compute.internal,}"
Oct 30 23:53:08.762516 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.762496    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ktxkb"
Oct 30 23:53:08.805670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.791574    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:53:08.805670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.798886    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:53:08.805670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.800378    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:53:08.805670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.800404    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:53:08.805670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.800427    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:53:08.805670 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.800461    2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:53:08.806039 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806022    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:08.806166 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806154    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:53:08.806226 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.806195    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:53:08.806226 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806207    2578 server.go:87] "Starting device plugin registration server"
Oct 30 23:53:08.806461 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806448    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:53:08.806526 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806462    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:53:08.806578 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806542    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:53:08.806642 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806625    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:53:08.806642 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.806639    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:53:08.807172 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.807155    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:53:08.807253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.807197    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:08.900973 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.900924    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"]
Oct 30 23:53:08.901059 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.900975    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.901770 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.901756    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.901848 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.901788    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.901848 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.901800    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.903988 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.903976    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.904183 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.904225 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904214    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.904668 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904654    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.904668 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904678    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.904792 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904688    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.904878 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904864    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.904922 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904897    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.904922 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.904913    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.906592 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.906573    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.906913 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.906900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.906977 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.906923    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:08.907269 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907254    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.907342 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907281    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.907342 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907295    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.907342 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907321    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.907665 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907650    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:08.907705 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907675    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:08.907705 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.907688    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:08.915983 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.915967    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.916052 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.915989    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-16.us-east-2.compute.internal\": node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:08.928051 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:08.926294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.929280 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.929261    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.931231 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.931214    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:08.952114 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:08.952095    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.027016 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.026997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.027110 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.027025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.027110 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.027043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe6922bb420b9fc1aa0231980b4f658c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"fe6922bb420b9fc1aa0231980b4f658c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.027193 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.027121    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.052273 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.052255    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.127285 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.127265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.127376 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.127288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe6922bb420b9fc1aa0231980b4f658c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"fe6922bb420b9fc1aa0231980b4f658c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.127376 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.127325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe6922bb420b9fc1aa0231980b4f658c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"fe6922bb420b9fc1aa0231980b4f658c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.127376 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.127363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.152436 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.152386    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.234066 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.234044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.237206 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.237187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.252681 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.252665    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.353100 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.353075    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.453664 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.453605    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.554096 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.554070    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:09.569086 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.569069    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:09.624787 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.624763    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:09.644764 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.644746    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:09.648870 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.648855    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 23:53:09.648967 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.648950    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:53:09.649010 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.648993    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:53:09.649010 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.648999    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:53:09.649067 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.649016    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:53:09.695405 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.695385    2578 apiserver.go:52] "Watching apiserver"
Oct 30 23:53:09.711708 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.711667    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:53:09.711997 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.711974    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bndnt","openshift-multus/multus-additional-cni-plugins-7bbnp","openshift-network-diagnostics/network-check-target-2qs9j","kube-system/global-pull-secret-syncer-ktssl","kube-system/konnectivity-agent-s8g6n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79","openshift-dns/node-resolver-ccfzb","openshift-multus/multus-4f6sj","openshift-multus/network-metrics-daemon-vl6sr","openshift-network-operator/iptables-alerter-jmmdd","openshift-ovn-kubernetes/ovnkube-node-hqmv6","openshift-cluster-node-tuning-operator/tuned-db4cd"]
Oct 30 23:53:09.715029 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.715015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.717443 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.717424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.718363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.718344    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:53:09.718363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.718344    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.718514 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.718348    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.718514 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.718449    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gq6b\""
Oct 30 23:53:09.719567 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.719553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:09.719623 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.719608    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:09.720014 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.720001    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:53:09.720345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.720329    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:53:09.720417 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.720344    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:53:09.720417 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.720344    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.720417 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.720377    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.720417 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.720378    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2wwf\""
Oct 30 23:53:09.722238 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.722222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.722299 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.722282    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:09.723944 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.723928    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 23:53:09.724294 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.724279    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.724407 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.724391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.726709 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.726694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.729022 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.729006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.730841 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.730821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.730936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.730856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-sys-fs\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.730936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.730875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-system-cni-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.730936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.730890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cnibin\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.731068 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.730957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-kube-api-access-859wz\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.731068 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.731068 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-konnectivity-ca\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.731165 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.731165 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-socket-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.731165 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-registration-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.731289 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e24b7db-f2a1-4500-9d82-963292cb7765-host\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.731289 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jpsf\" (UniqueName: \"kubernetes.io/projected/6e24b7db-f2a1-4500-9d82-963292cb7765-kube-api-access-4jpsf\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.731289 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.731424 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.731424 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/6b213ff9-3e00-426a-901f-e5ca105d37e8-kube-api-access-pqdzc\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.731424 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e24b7db-f2a1-4500-9d82-963292cb7765-serviceca\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.731424 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.731598 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:09.731598 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-kubelet-config\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.731598 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-dbus\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.731598 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-agent-certs\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.731598 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-device-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.731598 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.731568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.732336 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.732214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.735323 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.735303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:09.735408 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.735377    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:09.737856 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.737840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.740314 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.740299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.740938 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.740922    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:53:09.742616 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.742599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.742736 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.742714    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:53:09.742836 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.742785    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:09.743527 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.743510    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"]
Oct 30 23:53:09.761214 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.761190    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"]
Oct 30 23:53:09.761363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.761350    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:53:09.766563 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.766546    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p9sg2"
Oct 30 23:53:09.768677 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.768641    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:48:09 +0000 UTC" deadline="2028-05-02 02:25:00.865229412 +0000 UTC"
Oct 30 23:53:09.768757 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.768677    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13178h31m51.096556003s"
Oct 30 23:53:09.778572 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.778550    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p9sg2"
Oct 30 23:53:09.818867 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.818757    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:53:09.818867 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.818792    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.818867 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.818825    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:53:09.818867 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.818821    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.819063 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.818945    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.819063 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.818946    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:53:09.819063 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819009    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.819154 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819100    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.819192 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819170    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:53:09.819222 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819191    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:53:09.819259 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819226    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9xt4\""
Oct 30 23:53:09.819341 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819312    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.819453 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819377    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.819453 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819402    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z44sw\""
Oct 30 23:53:09.819623 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819474    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.819623 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819473    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tb4c7\""
Oct 30 23:53:09.819732 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819711    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wjrgq\""
Oct 30 23:53:09.819828 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819716    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:53:09.819828 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819751    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:53:09.819828 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819777    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q6gjg\""
Oct 30 23:53:09.819828 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819780    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-427wx\""
Oct 30 23:53:09.819973 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819919    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:53:09.819973 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819925    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:53:09.819973 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819936    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4mxgc\""
Oct 30 23:53:09.819973 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819962    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:53:09.819973 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.819972    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:53:09.825689 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.825674    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:53:09.832723 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-bin\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.832789 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-kubernetes\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.832789 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bbf2354-7b35-4374-adc3-e694b71f6e9e-hosts-file\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.832789 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.832921 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-system-cni-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.832921 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.832921 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.833021 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-multus\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.833021 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.832948    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-system-cni-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.833021 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-var-lib-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.833021 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.833013    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.833089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret podName:e73a9af6-0664-409b-a096-83a990ec98ce nodeName:}" failed. No retries permitted until 2026-10-30 23:53:10.33305484 +0000 UTC m=+2.027519693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret") pod "global-pull-secret-syncer-ktssl" (UID: "e73a9af6-0664-409b-a096-83a990ec98ce") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-kubelet-config\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e24b7db-f2a1-4500-9d82-963292cb7765-host\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833154    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e24b7db-f2a1-4500-9d82-963292cb7765-host\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-netns\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-kubelet-config\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.833196 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-hostroot\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-kubelet\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-node-log\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-config\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-modprobe-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e24b7db-f2a1-4500-9d82-963292cb7765-serviceca\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.833477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-k8s-cni-cncf-io\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.833701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-systemd\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.833701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.833701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-cni-binary-copy\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.833701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-iptables-alerter-script\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.833701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-run\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.833701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-tmp\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.833968 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.833947    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.834071 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.834124 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-kubelet\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.834124 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.834221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-multus-daemon-config\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.834221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e24b7db-f2a1-4500-9d82-963292cb7765-serviceca\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.834351 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-multus-certs\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.834403 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-host-slash\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.834450 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-ovn\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.834450 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-netd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.834540 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-script-lib\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.834540 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:09.834623 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-log-socket\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.834623 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysconfig\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.834709 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-var-lib-kubelet\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.834709 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-tuned\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.834792 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bbf2354-7b35-4374-adc3-e694b71f6e9e-tmp-dir\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.834792 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-system-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.834908 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8kf\" (UniqueName: \"kubernetes.io/projected/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-kube-api-access-cb8kf\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.834908 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-sys-fs\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.834908 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cnibin\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.834908 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:09.835074 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.834922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-systemd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835074 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-lib-modules\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.835074 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-konnectivity-ca\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.835074 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-socket-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.835241 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.835241 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-cnibin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.835241 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-slash\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835241 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-etc-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835241 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-host\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.835241 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrjb\" (UniqueName: \"kubernetes.io/projected/7057aab3-480b-4bd1-81cc-e100e33b35d5-kube-api-access-8vrjb\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.835507 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/6b213ff9-3e00-426a-901f-e5ca105d37e8-kube-api-access-pqdzc\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.835507 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-dbus\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.835507 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qs8\" (UniqueName: \"kubernetes.io/projected/5f6789dc-0c9b-4507-9615-93c3ec632af2-kube-api-access-t4qs8\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:09.835507 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835507 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-bin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovn-node-metrics-cert\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2rft\" (UniqueName: \"kubernetes.io/projected/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-kube-api-access-k2rft\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-kube-api-access-859wz\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-conf-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-env-overrides\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.835721 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-agent-certs\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-registration-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpsf\" (UniqueName: \"kubernetes.io/projected/6e24b7db-f2a1-4500-9d82-963292cb7765-kube-api-access-4jpsf\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dmv\" (UniqueName: \"kubernetes.io/projected/d1090e22-f04c-4bda-b340-0966209254af-kube-api-access-h2dmv\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-conf\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-sys\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.835980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-device-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgh87\" (UniqueName: \"kubernetes.io/projected/3bbf2354-7b35-4374-adc3-e694b71f6e9e-kube-api-access-sgh87\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.836036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-socket-dir-parent\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.836605 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-etc-kubernetes\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.836605 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-systemd-units\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.836605 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-netns\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.836605 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836475    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-sys-fs\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.836605 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cnibin\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.836955 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-registration-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.836955 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.836898    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:53:09.837106 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837049    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-device-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.837158 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837111    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.837866 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837432    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.837866 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837434    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-dbus\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:09.837866 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-konnectivity-ca\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.837866 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837559    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-socket-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.838215 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.837924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.840620 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.840592    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-agent-certs\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:09.849726 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.849648    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:53:09.849726 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.849667    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:53:09.849726 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.849676    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mhm77 for pod openshift-network-diagnostics/network-check-target-2qs9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:09.850027 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.849742    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77 podName:362b6ff8-9785-49c5-8704-db4c94a603d9 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:10.349725228 +0000 UTC m=+2.044190083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mhm77" (UniqueName: "kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77") pod "network-check-target-2qs9j" (UID: "362b6ff8-9785-49c5-8704-db4c94a603d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:09.851452 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.851432    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-kube-api-access-859wz\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:09.852501 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.852479    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpsf\" (UniqueName: \"kubernetes.io/projected/6e24b7db-f2a1-4500-9d82-963292cb7765-kube-api-access-4jpsf\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:09.852574 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.852556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/6b213ff9-3e00-426a-901f-e5ca105d37e8-kube-api-access-pqdzc\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:09.897960 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:09.897936    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d13e136b05ce96356f6b9a00f72419.slice/crio-b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09 WatchSource:0}: Error finding container b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09: Status 404 returned error can't find the container with id b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09
Oct 30 23:53:09.898498 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:09.898475    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6922bb420b9fc1aa0231980b4f658c.slice/crio-f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf WatchSource:0}: Error finding container f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf: Status 404 returned error can't find the container with id f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf
Oct 30 23:53:09.903711 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.903689    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:53:09.937025 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.936998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bbf2354-7b35-4374-adc3-e694b71f6e9e-tmp-dir\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.937025 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-system-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937192 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-system-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937192 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8kf\" (UniqueName: \"kubernetes.io/projected/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-kube-api-access-cb8kf\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.937192 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-systemd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-lib-modules\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-cnibin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.937256    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-slash\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-etc-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-host\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.937345 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bbf2354-7b35-4374-adc3-e694b71f6e9e-tmp-dir\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:09.937367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:10.437345349 +0000 UTC m=+2.131810189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-systemd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937391    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-host\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937395    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-lib-modules\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrjb\" (UniqueName: \"kubernetes.io/projected/7057aab3-480b-4bd1-81cc-e100e33b35d5-kube-api-access-8vrjb\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937424    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-etc-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937329    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-slash\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qs8\" (UniqueName: \"kubernetes.io/projected/5f6789dc-0c9b-4507-9615-93c3ec632af2-kube-api-access-t4qs8\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-cnibin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-bin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovn-node-metrics-cert\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937519    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rft\" (UniqueName: \"kubernetes.io/projected/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-kube-api-access-k2rft\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937551    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-bin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-conf-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.937669 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937589    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-conf-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-env-overrides\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937644    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dmv\" (UniqueName: \"kubernetes.io/projected/d1090e22-f04c-4bda-b340-0966209254af-kube-api-access-h2dmv\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-conf\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-sys\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgh87\" (UniqueName: \"kubernetes.io/projected/3bbf2354-7b35-4374-adc3-e694b71f6e9e-kube-api-access-sgh87\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937788    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-socket-dir-parent\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-etc-kubernetes\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937864    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-sys\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-conf\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.937989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-systemd-units\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-netns\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-bin\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.938506 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938050    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-socket-dir-parent\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-kubernetes\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-etc-kubernetes\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-systemd-units\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938095    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-bin\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bbf2354-7b35-4374-adc3-e694b71f6e9e-hosts-file\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938109    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-env-overrides\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-kubernetes\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bbf2354-7b35-4374-adc3-e694b71f6e9e-hosts-file\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-multus\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-netns\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938176    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-multus\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-var-lib-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-netns\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-hostroot\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-var-lib-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938288    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939096 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-kubelet\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-netns\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938324    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-kubelet\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-node-log\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-config\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938365    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-node-log\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938321    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-hostroot\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-modprobe-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-k8s-cni-cncf-io\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-systemd\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-cni-binary-copy\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-iptables-alerter-script\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938510    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-systemd\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-run\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938487    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-k8s-cni-cncf-io\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938468    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-modprobe-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-tmp\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-run\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.939701 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-kubelet\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-multus-daemon-config\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-multus-certs\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938660    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-kubelet\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-host-slash\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-ovn\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-netd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-script-lib\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-log-socket\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysconfig\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-var-lib-kubelet\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938848    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-multus-certs\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-tuned\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-host-slash\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysconfig\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.938973    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-log-socket\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940383 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.939002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-var-lib-kubelet\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.939036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-ovn\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.939066    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-netd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.939587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-config\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.939750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-cni-binary-copy\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.939791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-iptables-alerter-script\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.940091    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-multus-daemon-config\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.940936 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.940861    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-script-lib\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.941132 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.941018    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovn-node-metrics-cert\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.941396 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.941378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-tmp\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.941497 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.941482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-tuned\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.951141 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.951116    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8kf\" (UniqueName: \"kubernetes.io/projected/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-kube-api-access-cb8kf\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:09.952445 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.952423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrjb\" (UniqueName: \"kubernetes.io/projected/7057aab3-480b-4bd1-81cc-e100e33b35d5-kube-api-access-8vrjb\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:09.952681 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.952663    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dmv\" (UniqueName: \"kubernetes.io/projected/d1090e22-f04c-4bda-b340-0966209254af-kube-api-access-h2dmv\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:09.952857 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.952838    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgh87\" (UniqueName: \"kubernetes.io/projected/3bbf2354-7b35-4374-adc3-e694b71f6e9e-kube-api-access-sgh87\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:09.952901 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.952841    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rft\" (UniqueName: \"kubernetes.io/projected/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-kube-api-access-k2rft\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:09.955734 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:09.955713    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qs8\" (UniqueName: \"kubernetes.io/projected/5f6789dc-0c9b-4507-9615-93c3ec632af2-kube-api-access-t4qs8\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:10.058098 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.058045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:10.066147 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.066130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:10.067786 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.067766    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e24b7db_f2a1_4500_9d82_963292cb7765.slice/crio-2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556 WatchSource:0}: Error finding container 2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556: Status 404 returned error can't find the container with id 2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556
Oct 30 23:53:10.073738 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.073717    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1090e22_f04c_4bda_b340_0966209254af.slice/crio-dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527 WatchSource:0}: Error finding container dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527: Status 404 returned error can't find the container with id dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527
Oct 30 23:53:10.098317 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.098296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:10.102951 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.102933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:10.109926 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.109191    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd07a3a8_5fa2_41eb_afcd_4880f4f3cd58.slice/crio-c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9 WatchSource:0}: Error finding container c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9: Status 404 returned error can't find the container with id c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9
Oct 30 23:53:10.111540 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.111468    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fab40a_d3b0_4bf1_9ab3_396dfccebe5f.slice/crio-eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143 WatchSource:0}: Error finding container eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143: Status 404 returned error can't find the container with id eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143
Oct 30 23:53:10.113370 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.113356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:10.118005 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.117985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:10.122681 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.122662    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b213ff9_3e00_426a_901f_e5ca105d37e8.slice/crio-03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45 WatchSource:0}: Error finding container 03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45: Status 404 returned error can't find the container with id 03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45
Oct 30 23:53:10.123494 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.123479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:10.126549 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.126527    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbf2354_7b35_4374_adc3_e694b71f6e9e.slice/crio-543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3 WatchSource:0}: Error finding container 543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3: Status 404 returned error can't find the container with id 543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3
Oct 30 23:53:10.131630 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.131603    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa85bf8b_61bf_4bf1_b820_05215a7eadf3.slice/crio-e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11 WatchSource:0}: Error finding container e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11: Status 404 returned error can't find the container with id e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11
Oct 30 23:53:10.154537 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.154518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:10.161895 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.161875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:10.163745 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.163727    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057aab3_480b_4bd1_81cc_e100e33b35d5.slice/crio-d95579fcb6c64f0f98a67ee2e3a38931d7c11fb349a956c7b16f4db91b4af10d WatchSource:0}: Error finding container d95579fcb6c64f0f98a67ee2e3a38931d7c11fb349a956c7b16f4db91b4af10d: Status 404 returned error can't find the container with id d95579fcb6c64f0f98a67ee2e3a38931d7c11fb349a956c7b16f4db91b4af10d
Oct 30 23:53:10.170925 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:10.170906    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1a8190_b44f_41ac_8f0f_5020e6e27c18.slice/crio-cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c WatchSource:0}: Error finding container cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c: Status 404 returned error can't find the container with id cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c
Oct 30 23:53:10.342467 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.342395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:10.342595 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.342508    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:10.342595 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.342567    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret podName:e73a9af6-0664-409b-a096-83a990ec98ce nodeName:}" failed. No retries permitted until 2026-10-30 23:53:11.342550386 +0000 UTC m=+3.037015240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret") pod "global-pull-secret-syncer-ktssl" (UID: "e73a9af6-0664-409b-a096-83a990ec98ce") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:10.443071 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.443042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:10.443223 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.443088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:10.443298 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.443230    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:53:10.443298 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.443238    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:10.443298 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.443253    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:53:10.443298 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.443266    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mhm77 for pod openshift-network-diagnostics/network-check-target-2qs9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:10.443298 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.443291    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:11.443275991 +0000 UTC m=+3.137740832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:10.443552 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:10.443314    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77 podName:362b6ff8-9785-49c5-8704-db4c94a603d9 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:11.443300482 +0000 UTC m=+3.137765328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhm77" (UniqueName: "kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77") pod "network-check-target-2qs9j" (UID: "362b6ff8-9785-49c5-8704-db4c94a603d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:10.533346 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:10.533315    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:11.852831 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.852768    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:10 +0000 UTC" deadline="2028-04-29 00:23:16.712486085 +0000 UTC"
Oct 30 23:53:11.852831 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.852828    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13104h30m4.859661625s"
Oct 30 23:53:11.885568 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.885510    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-db4cd" event={"ID":"7057aab3-480b-4bd1-81cc-e100e33b35d5","Type":"ContainerStarted","Data":"d95579fcb6c64f0f98a67ee2e3a38931d7c11fb349a956c7b16f4db91b4af10d"}
Oct 30 23:53:11.898396 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.898353    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jmmdd" event={"ID":"fa85bf8b-61bf-4bf1-b820-05215a7eadf3","Type":"ContainerStarted","Data":"e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11"}
Oct 30 23:53:11.901019 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.900990    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" event={"ID":"6b213ff9-3e00-426a-901f-e5ca105d37e8","Type":"ContainerStarted","Data":"03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45"}
Oct 30 23:53:11.903229 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.903196    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bndnt" event={"ID":"6e24b7db-f2a1-4500-9d82-963292cb7765","Type":"ContainerStarted","Data":"2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556"}
Oct 30 23:53:11.906948 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.906923    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"f6d13e136b05ce96356f6b9a00f72419","Type":"ContainerStarted","Data":"b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09"}
Oct 30 23:53:11.909445 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.909417    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"fe6922bb420b9fc1aa0231980b4f658c","Type":"ContainerStarted","Data":"f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf"}
Oct 30 23:53:11.911267 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.911202    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c"}
Oct 30 23:53:11.918447 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.918387    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ccfzb" event={"ID":"3bbf2354-7b35-4374-adc3-e694b71f6e9e","Type":"ContainerStarted","Data":"543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3"}
Oct 30 23:53:11.932312 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.932286    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s8g6n" event={"ID":"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f","Type":"ContainerStarted","Data":"eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143"}
Oct 30 23:53:11.954839 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.954731    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerStarted","Data":"c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9"}
Oct 30 23:53:11.964633 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:11.964331    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4f6sj" event={"ID":"d1090e22-f04c-4bda-b340-0966209254af","Type":"ContainerStarted","Data":"dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527"}
Oct 30 23:53:12.415494 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.415403    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:12.420990 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.420958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:12.421148 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.421129    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:12.421214 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.421200    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret podName:e73a9af6-0664-409b-a096-83a990ec98ce nodeName:}" failed. No retries permitted until 2026-10-30 23:53:14.421178858 +0000 UTC m=+5.044987045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret") pod "global-pull-secret-syncer-ktssl" (UID: "e73a9af6-0664-409b-a096-83a990ec98ce") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:12.451571 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.451543    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:12.522191 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.522158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:12.522365 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.522212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:12.522427 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.522379    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:12.522499 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.522441    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:14.522421786 +0000 UTC m=+5.146229989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:12.522564 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.522554    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:53:12.522613 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.522576    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:53:12.522613 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.522588    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mhm77 for pod openshift-network-diagnostics/network-check-target-2qs9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:12.522708 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.522627    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77 podName:362b6ff8-9785-49c5-8704-db4c94a603d9 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:14.522614413 +0000 UTC m=+5.146422606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhm77" (UniqueName: "kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77") pod "network-check-target-2qs9j" (UID: "362b6ff8-9785-49c5-8704-db4c94a603d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:12.854065 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.853971    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:10 +0000 UTC" deadline="2028-06-25 11:42:24.68812084 +0000 UTC"
Oct 30 23:53:12.854065 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.854014    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14483h49m11.834112375s"
Oct 30 23:53:12.871324 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.871283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:12.871484 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.871415    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:12.871841 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.871822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:12.871960 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.871919    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:12.872012 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:12.871992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:12.872079 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:12.872061    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:13.116346 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:13.116283    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:14.436728 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:14.436685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:14.437190 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.436863    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:14.437190 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.436926    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret podName:e73a9af6-0664-409b-a096-83a990ec98ce nodeName:}" failed. No retries permitted until 2026-10-30 23:53:18.436907213 +0000 UTC m=+9.060715409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret") pod "global-pull-secret-syncer-ktssl" (UID: "e73a9af6-0664-409b-a096-83a990ec98ce") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:14.538742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:14.538798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.538950    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.539008    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:18.538991141 +0000 UTC m=+9.162799341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.539425    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.539444    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.539455    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mhm77 for pod openshift-network-diagnostics/network-check-target-2qs9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:14.539550 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.539519    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77 podName:362b6ff8-9785-49c5-8704-db4c94a603d9 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:18.539504728 +0000 UTC m=+9.163312913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhm77" (UniqueName: "kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77") pod "network-check-target-2qs9j" (UID: "362b6ff8-9785-49c5-8704-db4c94a603d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:14.871709 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:14.871650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:14.871916 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.871751    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:14.872089 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:14.872074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:14.872163 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.872148    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:14.872198 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:14.872193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:14.872260 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:14.872248    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:15.992489 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:15.991764    2578 generic.go:356] "Generic (PLEG): container finished" podID="f6d13e136b05ce96356f6b9a00f72419" containerID="d2633c913d8d14d20dc296c29382ae191fc6dfde0ae2c41de18a5b6671877c65" exitCode=0
Oct 30 23:53:15.992489 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:15.991918    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"f6d13e136b05ce96356f6b9a00f72419","Type":"ContainerDied","Data":"d2633c913d8d14d20dc296c29382ae191fc6dfde0ae2c41de18a5b6671877c65"}
Oct 30 23:53:15.998080 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:15.997603    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"fe6922bb420b9fc1aa0231980b4f658c","Type":"ContainerStarted","Data":"88a5b208d29e648818ff72025ce3d666e8ca81f32e51ca23218e8ff2824bfaed"}
Oct 30 23:53:16.002015 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.001450    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s8g6n" event={"ID":"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f","Type":"ContainerStarted","Data":"3e6fb3b5aaca9dbaf2680e4fcbfbc95207f51c751d9e5b5f6d469481348ea039"}
Oct 30 23:53:16.012004 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.011574    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="416dbee70618b84d1eb8e9b0696a20e6b1c0587c5582071e12199a2bc9796586" exitCode=0
Oct 30 23:53:16.012004 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.011611    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerDied","Data":"416dbee70618b84d1eb8e9b0696a20e6b1c0587c5582071e12199a2bc9796586"}
Oct 30 23:53:16.060637 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.060576    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal" podStartSLOduration=6.060559001 podStartE2EDuration="6.060559001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:16.060534693 +0000 UTC m=+6.684342896" watchObservedRunningTime="2026-10-30 23:53:16.060559001 +0000 UTC m=+6.684367201"
Oct 30 23:53:16.061798 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.061241    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s8g6n" podStartSLOduration=2.010183716 podStartE2EDuration="6.061226869s" totalImagesPullingTime="4.051043153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:16.042586166 +0000 UTC m=+6.666394368" watchObservedRunningTime="2026-10-30 23:53:16.061226869 +0000 UTC m=+6.685035075"
Oct 30 23:53:16.871320 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.871288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:16.871479 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:16.871416    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:16.871558 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.871506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:16.871615 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:16.871587    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:16.871668 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:16.871628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:16.871720 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:16.871681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:17.017363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:17.017284    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bndnt" event={"ID":"6e24b7db-f2a1-4500-9d82-963292cb7765","Type":"ContainerStarted","Data":"7bf5e4f3a42eb6c21b7478dc977bbcfe3d8db3ddc9660e1a7eff3dd057216f7d"}
Oct 30 23:53:17.025153 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:17.025126    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"f6d13e136b05ce96356f6b9a00f72419","Type":"ContainerStarted","Data":"cdf6b0bcba652d08360b814c0c6c9c0c65f820d0fc5cfba5b01c982e8e61e0a1"}
Oct 30 23:53:17.038060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:17.038008    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bndnt" podStartSLOduration=1.53291215 podStartE2EDuration="7.037990885s" totalImagesPullingTime="5.505078735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:17.037313478 +0000 UTC m=+7.661121691" watchObservedRunningTime="2026-10-30 23:53:17.037990885 +0000 UTC m=+7.661799084"
Oct 30 23:53:18.031831 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.031802    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" event={"ID":"6b213ff9-3e00-426a-901f-e5ca105d37e8","Type":"ContainerStarted","Data":"458b8de1eabbd4fb0434a14ff4635e775710a437176aa0004b218e0563c1be9c"}
Oct 30 23:53:18.392640 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.392562    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:18.393818 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.393795    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:18.416091 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.416029    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" podStartSLOduration=8.416013328 podStartE2EDuration="8.416013328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:17.052973864 +0000 UTC m=+7.676782066" watchObservedRunningTime="2026-10-30 23:53:18.416013328 +0000 UTC m=+9.039821719"
Oct 30 23:53:18.468059 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.468023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:18.469167 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.468643    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:18.469167 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.468721    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret podName:e73a9af6-0664-409b-a096-83a990ec98ce nodeName:}" failed. No retries permitted until 2026-10-30 23:53:26.468700206 +0000 UTC m=+17.092508400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret") pod "global-pull-secret-syncer-ktssl" (UID: "e73a9af6-0664-409b-a096-83a990ec98ce") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.569019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.569076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.569278    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.569301    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.569315    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mhm77 for pod openshift-network-diagnostics/network-check-target-2qs9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.569379    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77 podName:362b6ff8-9785-49c5-8704-db4c94a603d9 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:26.569360318 +0000 UTC m=+17.193168516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhm77" (UniqueName: "kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77") pod "network-check-target-2qs9j" (UID: "362b6ff8-9785-49c5-8704-db4c94a603d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.569776    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:18.569891 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.569834    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:26.569815098 +0000 UTC m=+17.193623283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:18.871478 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.871431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:18.871615 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.871575    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:18.872119 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.871925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:18.872119 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.872003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:18.872119 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.872038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:18.872119 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:18.872083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:18.892590 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:18.892561    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:53:19.036276 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:19.036167    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" event={"ID":"6b213ff9-3e00-426a-901f-e5ca105d37e8","Type":"ContainerStarted","Data":"9a00753939905009d3070628266c86e861ad6be971be7b186fd05c52259e34c2"}
Oct 30 23:53:19.883898 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:19.883758    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:53:18.892576397Z","UUID":"d2bc7a16-1b00-4274-9794-142a9795f7ec","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:53:19.885412 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:19.885237    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:53:19.885552 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:19.885434    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:53:20.040004 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.039955    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ccfzb" event={"ID":"3bbf2354-7b35-4374-adc3-e694b71f6e9e","Type":"ContainerStarted","Data":"997f78a75e92554138095221c20d6f631c9529e0119190e44bc45ccacf8f1ddd"}
Oct 30 23:53:20.041888 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.041861    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4f6sj" event={"ID":"d1090e22-f04c-4bda-b340-0966209254af","Type":"ContainerStarted","Data":"60758e0105cda742a7f9b77e2c4a3af03fcfc65a950aa172975edcdd7e8a5bb4"}
Oct 30 23:53:20.057906 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.057858    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ccfzb" podStartSLOduration=1.6559107370000001 podStartE2EDuration="10.057844588s" totalImagesPullingTime="8.401933851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:20.057719261 +0000 UTC m=+10.681527518" watchObservedRunningTime="2026-10-30 23:53:20.057844588 +0000 UTC m=+10.681652792"
Oct 30 23:53:20.081729 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.081686    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4f6sj" podStartSLOduration=1.947838413 podStartE2EDuration="10.081672922s" totalImagesPullingTime="8.133834509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:20.081133326 +0000 UTC m=+10.704941558" watchObservedRunningTime="2026-10-30 23:53:20.081672922 +0000 UTC m=+10.705481124"
Oct 30 23:53:20.871691 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.871655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:20.871842 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:20.871767    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:20.872153 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.872130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:20.872236 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:20.872220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:20.872278 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:20.872272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:20.872355 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:20.872331    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:21.046228 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:21.046187    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="242dcf40320fd6ec74a851427390243a49a79fc4c004d8eec21da970a39c3276" exitCode=0
Oct 30 23:53:21.046630 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:21.046288    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerDied","Data":"242dcf40320fd6ec74a851427390243a49a79fc4c004d8eec21da970a39c3276"}
Oct 30 23:53:21.049864 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:21.049841    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" event={"ID":"6b213ff9-3e00-426a-901f-e5ca105d37e8","Type":"ContainerStarted","Data":"382ae3ab4671df0c6445eda8ceb5da78b33bcdf53a5a9734cffde76b30278fad"}
Oct 30 23:53:21.090976 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:21.090939    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podStartSLOduration=2.570674891 podStartE2EDuration="11.090923813s" totalImagesPullingTime="8.520248922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:21.090600019 +0000 UTC m=+11.714408223" watchObservedRunningTime="2026-10-30 23:53:21.090923813 +0000 UTC m=+11.714732021"
Oct 30 23:53:22.052480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.052432    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-db4cd" event={"ID":"7057aab3-480b-4bd1-81cc-e100e33b35d5","Type":"ContainerStarted","Data":"14a1774a077d75f06cd64c1376d0a66769c52bd8df971796b09b313c1d3ac34c"}
Oct 30 23:53:22.053968 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.053937    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jmmdd" event={"ID":"fa85bf8b-61bf-4bf1-b820-05215a7eadf3","Type":"ContainerStarted","Data":"bd7af489275f2f27e4b4bc05b77d3f815f71a6c3905780a046f4f5df4364ceb2"}
Oct 30 23:53:22.072267 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.072227    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-db4cd" podStartSLOduration=1.81146736 podStartE2EDuration="12.072215425s" totalImagesPullingTime="10.260748065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:22.072212381 +0000 UTC m=+12.696020595" watchObservedRunningTime="2026-10-30 23:53:22.072215425 +0000 UTC m=+12.696023627"
Oct 30 23:53:22.099495 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.099435    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jmmdd" podStartSLOduration=3.598597636 podStartE2EDuration="12.099422579s" totalImagesPullingTime="8.500824943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:22.099122341 +0000 UTC m=+12.722930544" watchObservedRunningTime="2026-10-30 23:53:22.099422579 +0000 UTC m=+12.723230781"
Oct 30 23:53:22.871641 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.871609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:22.871641 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.871639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:22.871849 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:22.871735    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:22.871900 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:22.871831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:22.871993 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:22.871965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:22.872133 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:22.872052    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:23.057606 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:23.057578    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="e614d2eafab726af5128ab3b28a5a1e4177e1283b7ac3d16950c92324f340048" exitCode=0
Oct 30 23:53:23.058094 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:23.057666    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerDied","Data":"e614d2eafab726af5128ab3b28a5a1e4177e1283b7ac3d16950c92324f340048"}
Oct 30 23:53:24.628526 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:24.628238    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:24.629013 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:24.628575    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:53:24.629251 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:24.629223    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:24.871664 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:24.871635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:24.871817 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:24.871665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:24.871817 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:24.871753    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:24.871817 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:24.871817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:24.872021 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:24.871928    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:24.872021 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:24.872012    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:25.064396 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:25.064323    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="3354262f7bcf0e8ac775cd5ac943ea41c4fad62443f091995bd521b9aaaeb5a7" exitCode=0
Oct 30 23:53:25.064563 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:25.064419    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerDied","Data":"3354262f7bcf0e8ac775cd5ac943ea41c4fad62443f091995bd521b9aaaeb5a7"}
Oct 30 23:53:26.537389 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:26.537350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:26.537872 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.537529    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:26.537872 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.537607    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret podName:e73a9af6-0664-409b-a096-83a990ec98ce nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.537586846 +0000 UTC m=+33.161395046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret") pod "global-pull-secret-syncer-ktssl" (UID: "e73a9af6-0664-409b-a096-83a990ec98ce") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:53:26.638391 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:26.638360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:26.638391 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:26.638404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:26.638647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.638534    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:26.638647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.638548    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:53:26.638647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.638573    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:53:26.638647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.638587    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mhm77 for pod openshift-network-diagnostics/network-check-target-2qs9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:26.638647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.638594    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.638576209 +0000 UTC m=+33.262384405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:53:26.638647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.638633    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77 podName:362b6ff8-9785-49c5-8704-db4c94a603d9 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.638621793 +0000 UTC m=+33.262429987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhm77" (UniqueName: "kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77") pod "network-check-target-2qs9j" (UID: "362b6ff8-9785-49c5-8704-db4c94a603d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:53:26.872316 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:26.871800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:26.872316 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:26.871917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:26.872316 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.871926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:26.872316 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:26.871957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:26.872316 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.872053    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:26.872316 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:26.872160    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:28.073357 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.073179    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"312b8af1b20ca0eba4d24a22efe9efe46aa55feea1005644b5ff20772323b1d4"}
Oct 30 23:53:28.073713 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.073395    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"41d3396d39cfd0f6e988a4e1577a5f6eb48187b4db4f5e980711c653582db3bd"}
Oct 30 23:53:28.073713 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.073439    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"ec67b477f612bc00306acf3b98e2d10bc6759db495dacccb36805f5d48368cbf"}
Oct 30 23:53:28.076180 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.076158    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="58743ea02cf0a206407ea8dd831f449b2aba136716415162f28b9a19612626d4" exitCode=0
Oct 30 23:53:28.076246 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.076204    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerDied","Data":"58743ea02cf0a206407ea8dd831f449b2aba136716415162f28b9a19612626d4"}
Oct 30 23:53:28.871977 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.871933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:28.871977 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.871965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:28.872199 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:28.872029    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:28.872199 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:28.872062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:28.872199 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:28.872137    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:28.872325 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:28.872226    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:29.081395 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:29.081359    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"31267479a9426eeab1cdbc6d3ccaaa15862523c43b138e0f5ab8e1da10057074"}
Oct 30 23:53:29.081933 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:29.081440    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"d67a37cb817e421298a0ffdf69b33e6155ee83766e5ce6af1df21fd16d1533f7"}
Oct 30 23:53:29.081933 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:29.081493    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"88aea85719923e4dee7226d5c734ead2fe0b717f6e7ea216e88bcfce6e9dde61"}
Oct 30 23:53:29.084359 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:29.084331    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="b80be36abb6e7ca0b6b972aa77c6b7b69c34e354b38cd327ba686aa605530aff" exitCode=0
Oct 30 23:53:29.084475 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:29.084371    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerDied","Data":"b80be36abb6e7ca0b6b972aa77c6b7b69c34e354b38cd327ba686aa605530aff"}
Oct 30 23:53:30.088885 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:30.088850    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" event={"ID":"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58","Type":"ContainerStarted","Data":"14c99ae138dc326723b44b9d99b0af35b2ed9b32b2234628c899f0b63804e5a4"}
Oct 30 23:53:30.113043 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:30.112999    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7bbnp" podStartSLOduration=8.10701917 podStartE2EDuration="20.112987489s" totalImagesPullingTime="12.005968319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:30.112045761 +0000 UTC m=+20.735853962" watchObservedRunningTime="2026-10-30 23:53:30.112987489 +0000 UTC m=+20.736795690"
Oct 30 23:53:30.871639 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:30.871611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:30.871785 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:30.871661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:30.871785 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:30.871672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:30.871785 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:30.871749    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:30.871886 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:30.871817    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:30.871922 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:30.871893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:31.095723 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:31.094019    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"aa76c00dd70909e2a77144976dd75f3a95820f0a35746be52737a0534843b95d"}
Oct 30 23:53:32.871311 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:32.871281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:32.871911 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:32.871379    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:32.871911 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:32.871394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:32.871911 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:32.871294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:32.871911 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:32.871492    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:32.871911 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:32.871572    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:33.100895 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:33.100863    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" event={"ID":"1a1a8190-b44f-41ac-8f0f-5020e6e27c18","Type":"ContainerStarted","Data":"37bb93a80c0f02293ad10905626ee1a2a3050ba6d319379f82571c346b33c9ce"}
Oct 30 23:53:33.101152 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:33.101138    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:33.117477 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:33.117440    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:33.132769 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:33.132688    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" podStartSLOduration=6.476596228 podStartE2EDuration="23.132673299s" totalImagesPullingTime="16.656077071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:33.131551753 +0000 UTC m=+23.755359955" watchObservedRunningTime="2026-10-30 23:53:33.132673299 +0000 UTC m=+23.756481500"
Oct 30 23:53:34.104431 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.104227    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:34.104878 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.104512    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:34.121056 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.121030    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:34.129352 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.129328    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ktssl"]
Oct 30 23:53:34.129492 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.129443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:34.129575 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:34.129557    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:34.133538 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.133516    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qs9j"]
Oct 30 23:53:34.133658 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.133621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:34.133737 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:34.133714    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:34.134154 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.134136    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vl6sr"]
Oct 30 23:53:34.134250 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:34.134236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:34.134354 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:34.134336    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:35.872003 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:35.871972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:35.872508 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:35.871971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:35.872508 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:35.872087    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:35.872508 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:35.871971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:35.872508 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:35.872140    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:35.872508 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:35.872212    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:37.871969 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:37.871940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:37.871969 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:37.871966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:37.872398 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:37.872030    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:37.872398 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:37.872084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:37.872398 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:37.872194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:37.872398 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:37.872234    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:39.872520 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:39.872492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:39.872929 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:39.872604    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktssl" podUID="e73a9af6-0664-409b-a096-83a990ec98ce"
Oct 30 23:53:39.872929 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:39.872689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:39.872929 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:39.872796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:39.872929 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:39.872821    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9"
Oct 30 23:53:39.872929 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:39.872889    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vl6sr" podUID="5f6789dc-0c9b-4507-9615-93c3ec632af2"
Oct 30 23:53:40.106796 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.106726    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeReady"
Oct 30 23:53:40.106932 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.106882    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 23:53:40.144915 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.144886    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69tlv"]
Oct 30 23:53:40.196542 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.196492    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kgs9w"]
Oct 30 23:53:40.196717 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.196550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:40.199844 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.199820    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.200600 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.200565    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.211509 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.211486    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2qk9k"]
Oct 30 23:53:40.211595 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.211546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:40.225601 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.225580    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"]
Oct 30 23:53:40.225710 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.225695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:40.250916 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.250852    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mb6hw"]
Oct 30 23:53:40.251025 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.250945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.255006 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.254986    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 23:53:40.255129 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.255112    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bdhrq\""
Oct 30 23:53:40.259737 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.259721    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74cc75bf99-fkb87"]
Oct 30 23:53:40.259917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.259900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.262301 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.262283    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 23:53:40.262641 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.262624    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q4t65\""
Oct 30 23:53:40.264635 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.264619    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"]
Oct 30 23:53:40.264785 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.264768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.277422 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.277405    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"]
Oct 30 23:53:40.277528 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.277506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.300917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300822    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.300917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300839    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 23:53:40.300917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300843    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.300917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300826    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 23:53:40.300917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300859    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.300917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300905    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.301234 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300951    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.301234 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300962    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:53:40.301234 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.300979    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.301234 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.301013    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.301234 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.301022    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.301578 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.301555    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674dd678bd-xdv8h"]
Oct 30 23:53:40.301731 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.301710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.302420 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.302405    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 23:53:40.302899 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.302878    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:53:40.309223 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.309207    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"]
Oct 30 23:53:40.309355 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.309340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.315544 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.315528    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"]
Oct 30 23:53:40.315649 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.315637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.318903 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.318884    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"]
Oct 30 23:53:40.319283 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.319271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:40.323766 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.323741    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6449bcf677-rqxsk"]
Oct 30 23:53:40.323872 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.323834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:40.341593 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-stats-auth\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.341682 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.341682 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-snapshots\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.341682 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpwz\" (UniqueName: \"kubernetes.io/projected/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-kube-api-access-fgpwz\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.341815 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8586\" (UniqueName: \"kubernetes.io/projected/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-kube-api-access-g8586\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.341855 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-serving-cert\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.341855 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8h7\" (UniqueName: \"kubernetes.io/projected/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-kube-api-access-fl8h7\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.341927 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4bc\" (UniqueName: \"kubernetes.io/projected/be5dad9f-114f-4b12-bd51-217374c94fc0-kube-api-access-nt4bc\") pod \"kas-connection-checker-685895c7c9-2qk9k\" (UID: \"be5dad9f-114f-4b12-bd51-217374c94fc0\") " pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:40.341979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-default-certificate\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.341979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.341951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.342074 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.342126 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwq4\" (UniqueName: \"kubernetes.io/projected/25170a0c-24d2-48b8-80fa-d2d2780a41a3-kube-api-access-xkwq4\") pod \"kas-connection-checker-685895c7c9-kgs9w\" (UID: \"25170a0c-24d2-48b8-80fa-d2d2780a41a3\") " pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:40.342178 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.342178 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.342267 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwggr\" (UniqueName: \"kubernetes.io/projected/51bd9499-546b-4b4e-8350-7501c25b0ab2-kube-api-access-qwggr\") pod \"kas-connection-checker-685895c7c9-69tlv\" (UID: \"51bd9499-546b-4b4e-8350-7501c25b0ab2\") " pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:40.342296 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-tmp\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.342296 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.342285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-service-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.349181 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.349158    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69tlv"]
Oct 30 23:53:40.349247 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.349186    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-shskm"]
Oct 30 23:53:40.349288 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.349260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.355635 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355622    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 23:53:40.355717 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355661    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 23:53:40.355870 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355852    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 23:53:40.355921 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355881    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 23:53:40.355921 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355905    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 23:53:40.355921 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355907    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355927    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355935    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355935    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8s8qp\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355980    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z8jlp\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355981    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r8slq\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355995    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vb9sd\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.355997    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.356016    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7dvpv\""
Oct 30 23:53:40.356060 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.356023    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 23:53:40.370291 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.370248    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"]
Oct 30 23:53:40.370363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.370354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.385379 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.385362    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kgs9w"]
Oct 30 23:53:40.385448 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.385391    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mrwcz"]
Oct 30 23:53:40.385507 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.385492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.399624 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.399607    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"]
Oct 30 23:53:40.399738 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.399724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.404711 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404640    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.404711 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404648    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.404883 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404778    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 23:53:40.404883 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404841    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.405005 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404893    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 23:53:40.405005 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404925    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:53:40.405005 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.404949    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 23:53:40.405166 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.405012    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.405166 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.405044    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.405166 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.405144    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.405864 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.405848    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.405944 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.405882    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.405993 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.405955    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 23:53:40.408243 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.408227    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 23:53:40.408426 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.408412    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 23:53:40.419199 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419183    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"]
Oct 30 23:53:40.419248 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419211    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2qk9k"]
Oct 30 23:53:40.419248 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419232    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"]
Oct 30 23:53:40.419312 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419278    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mb6hw"]
Oct 30 23:53:40.419312 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419301    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"]
Oct 30 23:53:40.419369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:40.419369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419327    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674dd678bd-xdv8h"]
Oct 30 23:53:40.419369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419347    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"]
Oct 30 23:53:40.419369 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419367    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"]
Oct 30 23:53:40.419515 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419388    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"]
Oct 30 23:53:40.419515 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419407    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-shskm"]
Oct 30 23:53:40.419515 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419431    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"]
Oct 30 23:53:40.419515 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419450    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"]
Oct 30 23:53:40.419515 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419487    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74cc75bf99-fkb87"]
Oct 30 23:53:40.419515 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419507    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrwcz"]
Oct 30 23:53:40.419670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.419526    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6449bcf677-rqxsk"]
Oct 30 23:53:40.443019 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.442996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbp2v\" (UniqueName: \"kubernetes.io/projected/b25a3f44-1814-43a8-a01d-f98c870b7d9f-kube-api-access-tbp2v\") pod \"volume-data-source-validator-58c9f5f46d-sf4kv\" (UID: \"b25a3f44-1814-43a8-a01d-f98c870b7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:40.443121 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8586\" (UniqueName: \"kubernetes.io/projected/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-kube-api-access-g8586\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.443121 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.443121 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8h7\" (UniqueName: \"kubernetes.io/projected/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-kube-api-access-fl8h7\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.443280 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.443280 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.443280 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-default-certificate\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.443280 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.443280 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmm2\" (UniqueName: \"kubernetes.io/projected/a286ab8f-3f4f-4621-8b10-3730b4e114f7-kube-api-access-dhmm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.443280 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf52a97-c6ca-4fba-a77b-67841614c162-config\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.443611 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.443611 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.443611 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.443452    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 23:53:40.443611 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.443611 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.443524    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:40.943508903 +0000 UTC m=+31.567317084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : secret "router-metrics-certs-default" not found
Oct 30 23:53:40.443611 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b949609-5560-4b48-a5b7-229d7430d7c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:40.443923 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.443903    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.444102 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.444179 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.444179 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.444277 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.444185    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:40.944167249 +0000 UTC m=+31.567975448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:40.444277 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.444277 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a286ab8f-3f4f-4621-8b10-3730b4e114f7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.444372 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-stats-auth\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.444372 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6w5v\" (UniqueName: \"kubernetes.io/projected/daf52a97-c6ca-4fba-a77b-67841614c162-kube-api-access-x6w5v\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.444372 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.444538 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.444538 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf52a97-c6ca-4fba-a77b-67841614c162-serving-cert\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.444538 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjsr\" (UniqueName: \"kubernetes.io/projected/964f2946-7176-4f03-b41f-03bd65aa8346-kube-api-access-pqjsr\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.444538 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a286ab8f-3f4f-4621-8b10-3730b4e114f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.444538 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-serving-cert\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.444783 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4bc\" (UniqueName: \"kubernetes.io/projected/be5dad9f-114f-4b12-bd51-217374c94fc0-kube-api-access-nt4bc\") pod \"kas-connection-checker-685895c7c9-2qk9k\" (UID: \"be5dad9f-114f-4b12-bd51-217374c94fc0\") " pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:40.444783 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.444783 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.444783 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.444783 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.444783 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwggr\" (UniqueName: \"kubernetes.io/projected/51bd9499-546b-4b4e-8350-7501c25b0ab2-kube-api-access-qwggr\") pod \"kas-connection-checker-685895c7c9-69tlv\" (UID: \"51bd9499-546b-4b4e-8350-7501c25b0ab2\") " pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwq4\" (UniqueName: \"kubernetes.io/projected/25170a0c-24d2-48b8-80fa-d2d2780a41a3-kube-api-access-xkwq4\") pod \"kas-connection-checker-685895c7c9-kgs9w\" (UID: \"25170a0c-24d2-48b8-80fa-d2d2780a41a3\") " pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.444928    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.444968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-tmp\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.444996    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:40.944980147 +0000 UTC m=+31.568788328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-service-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-config-volume\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-tmp-dir\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.445388 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95p8n\" (UniqueName: \"kubernetes.io/projected/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-kube-api-access-95p8n\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.446200 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445752    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.446200 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.446200 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-tmp\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.446200 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-snapshots\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.446200 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.445912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpwz\" (UniqueName: \"kubernetes.io/projected/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-kube-api-access-fgpwz\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.446643 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.446622    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-snapshots\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.446843 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.446823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-service-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.450862 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.450838    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-serving-cert\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.450954 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.450900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-default-certificate\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.450954 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.450906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-stats-auth\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.455781 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.455756    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2gpl5\""
Oct 30 23:53:40.455908 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.455892    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhccc\""
Oct 30 23:53:40.456031 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456012    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4xs9v\""
Oct 30 23:53:40.456112 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456096    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:53:40.456181 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456168    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 23:53:40.456235 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8586\" (UniqueName: \"kubernetes.io/projected/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-kube-api-access-g8586\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.456235 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456201    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:53:40.456834 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456797    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8h7\" (UniqueName: \"kubernetes.io/projected/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-kube-api-access-fl8h7\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.456957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.456939    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwggr\" (UniqueName: \"kubernetes.io/projected/51bd9499-546b-4b4e-8350-7501c25b0ab2-kube-api-access-qwggr\") pod \"kas-connection-checker-685895c7c9-69tlv\" (UID: \"51bd9499-546b-4b4e-8350-7501c25b0ab2\") " pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:40.458130 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.458110    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpwz\" (UniqueName: \"kubernetes.io/projected/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-kube-api-access-fgpwz\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.458577 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.458556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4bc\" (UniqueName: \"kubernetes.io/projected/be5dad9f-114f-4b12-bd51-217374c94fc0-kube-api-access-nt4bc\") pod \"kas-connection-checker-685895c7c9-2qk9k\" (UID: \"be5dad9f-114f-4b12-bd51-217374c94fc0\") " pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:40.459315 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.459294    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwq4\" (UniqueName: \"kubernetes.io/projected/25170a0c-24d2-48b8-80fa-d2d2780a41a3-kube-api-access-xkwq4\") pod \"kas-connection-checker-685895c7c9-kgs9w\" (UID: \"25170a0c-24d2-48b8-80fa-d2d2780a41a3\") " pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:40.502648 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.502614    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.502648 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.502636    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.502648 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.502617    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.502851 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.502614    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.520308 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.520285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:40.547958 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.547901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:40.548109 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.548221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:40.548221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.548221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-config-volume\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.548221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.548221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-tmp-dir\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95p8n\" (UniqueName: \"kubernetes.io/projected/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-kube-api-access-95p8n\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbp2v\" (UniqueName: \"kubernetes.io/projected/b25a3f44-1814-43a8-a01d-f98c870b7d9f-kube-api-access-tbp2v\") pod \"volume-data-source-validator-58c9f5f46d-sf4kv\" (UID: \"b25a3f44-1814-43a8-a01d-f98c870b7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548302    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548304    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvx2l\" (UniqueName: \"kubernetes.io/projected/7eb92403-d07f-4bc4-bf34-4a15adff4863-kube-api-access-fvx2l\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548362    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls podName:964f2946-7176-4f03-b41f-03bd65aa8346 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.048342083 +0000 UTC m=+31.672150267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h2j9x" (UID: "964f2946-7176-4f03-b41f-03bd65aa8346") : secret "samples-operator-tls" not found
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548396    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:53:40.548438 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548441    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert podName:1b949609-5560-4b48-a5b7-229d7430d7c0 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.048421168 +0000 UTC m=+31.672229364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-k7rhq" (UID: "1b949609-5560-4b48-a5b7-229d7430d7c0") : secret "networking-console-plugin-cert" not found
Oct 30 23:53:40.548893 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548568    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls podName:d1384f8a-a2ad-4e55-b6e5-14ab134fa489 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.048548861 +0000 UTC m=+31.672357054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls") pod "dns-default-shskm" (UID: "d1384f8a-a2ad-4e55-b6e5-14ab134fa489") : secret "dns-default-metrics-tls" not found
Oct 30 23:53:40.548893 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.548893 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.548893 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.548893 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-tmp-dir\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmm2\" (UniqueName: \"kubernetes.io/projected/a286ab8f-3f4f-4621-8b10-3730b4e114f7-kube-api-access-dhmm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548967    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.548976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf52a97-c6ca-4fba-a77b-67841614c162-config\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.548980    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-config-volume\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.549038    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.049024676 +0000 UTC m=+31.672832873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549054    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.549135 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4557\" (UniqueName: \"kubernetes.io/projected/fe484ab6-ac95-4849-acc0-37636302d300-kube-api-access-r4557\") pod \"network-check-source-6f78d59f99-rw5ms\" (UID: \"fe484ab6-ac95-4849-acc0-37636302d300\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b949609-5560-4b48-a5b7-229d7430d7c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-config\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-serving-cert\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a286ab8f-3f4f-4621-8b10-3730b4e114f7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549511    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf52a97-c6ca-4fba-a77b-67841614c162-config\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6w5v\" (UniqueName: \"kubernetes.io/projected/daf52a97-c6ca-4fba-a77b-67841614c162-kube-api-access-x6w5v\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.549582 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb92403-d07f-4bc4-bf34-4a15adff4863-cert\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf52a97-c6ca-4fba-a77b-67841614c162-serving-cert\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tst\" (UniqueName: \"kubernetes.io/projected/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-kube-api-access-v8tst\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjsr\" (UniqueName: \"kubernetes.io/projected/964f2946-7176-4f03-b41f-03bd65aa8346-kube-api-access-pqjsr\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a286ab8f-3f4f-4621-8b10-3730b4e114f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.549980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.550150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.550139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b949609-5560-4b48-a5b7-229d7430d7c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:40.550779 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.550445    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.551375 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.551351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.551806 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.551779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.551902 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.551876    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:40.551902 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.551890    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:40.552013 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.551944    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.051928334 +0000 UTC m=+31.675736536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:40.552013 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.551883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.552115 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.552063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.552513 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.552488    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a286ab8f-3f4f-4621-8b10-3730b4e114f7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.552588 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.552528    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.553129 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.553104    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a286ab8f-3f4f-4621-8b10-3730b4e114f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.553200 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.553137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.553328 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.553306    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.553866 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.553844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.554236 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.554216    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf52a97-c6ca-4fba-a77b-67841614c162-serving-cert\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.559042 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.559026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:40.563038 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.563012    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95p8n\" (UniqueName: \"kubernetes.io/projected/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-kube-api-access-95p8n\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:40.563509 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.563484    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.563607 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.563511    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.564389 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.564360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbp2v\" (UniqueName: \"kubernetes.io/projected/b25a3f44-1814-43a8-a01d-f98c870b7d9f-kube-api-access-tbp2v\") pod \"volume-data-source-validator-58c9f5f46d-sf4kv\" (UID: \"b25a3f44-1814-43a8-a01d-f98c870b7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:40.564518 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.564401    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:40.564756 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.564737    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6w5v\" (UniqueName: \"kubernetes.io/projected/daf52a97-c6ca-4fba-a77b-67841614c162-kube-api-access-x6w5v\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.564823 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.564761    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:40.565380 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.565360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjsr\" (UniqueName: \"kubernetes.io/projected/964f2946-7176-4f03-b41f-03bd65aa8346-kube-api-access-pqjsr\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:40.566317 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.566301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmm2\" (UniqueName: \"kubernetes.io/projected/a286ab8f-3f4f-4621-8b10-3730b4e114f7-kube-api-access-dhmm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.582284 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.582262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:40.614650 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.614589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:40.650432 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4557\" (UniqueName: \"kubernetes.io/projected/fe484ab6-ac95-4849-acc0-37636302d300-kube-api-access-r4557\") pod \"network-check-source-6f78d59f99-rw5ms\" (UID: \"fe484ab6-ac95-4849-acc0-37636302d300\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:40.650697 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-config\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.650697 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-serving-cert\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.650697 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb92403-d07f-4bc4-bf34-4a15adff4863-cert\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.650697 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tst\" (UniqueName: \"kubernetes.io/projected/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-kube-api-access-v8tst\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.650942 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvx2l\" (UniqueName: \"kubernetes.io/projected/7eb92403-d07f-4bc4-bf34-4a15adff4863-kube-api-access-fvx2l\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.651017 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.650965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.651930 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.651897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-config\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.652018 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.651988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.655290 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.655255    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-serving-cert\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.656823 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.656797    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb92403-d07f-4bc4-bf34-4a15adff4863-cert\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.657494 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.657423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:40.663806 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.663705    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4557\" (UniqueName: \"kubernetes.io/projected/fe484ab6-ac95-4849-acc0-37636302d300-kube-api-access-r4557\") pod \"network-check-source-6f78d59f99-rw5ms\" (UID: \"fe484ab6-ac95-4849-acc0-37636302d300\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:40.664358 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.664336    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvx2l\" (UniqueName: \"kubernetes.io/projected/7eb92403-d07f-4bc4-bf34-4a15adff4863-kube-api-access-fvx2l\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.675424 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.671102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tst\" (UniqueName: \"kubernetes.io/projected/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-kube-api-access-v8tst\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.700255 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.700220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:40.783963 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.783938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:40.831125 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.831094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:40.858753 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.855845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:40.858753 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.858435    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kgs9w"]
Oct 30 23:53:40.858753 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.858554    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69tlv"]
Oct 30 23:53:40.895998 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.895260    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2qk9k"]
Oct 30 23:53:40.905372 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:40.904628    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5dad9f_114f_4b12_bd51_217374c94fc0.slice/crio-0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2 WatchSource:0}: Error finding container 0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2: Status 404 returned error can't find the container with id 0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2
Oct 30 23:53:40.934529 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.933887    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mb6hw"]
Oct 30 23:53:40.946684 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.946599    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"]
Oct 30 23:53:40.953979 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.953925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.954090 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.954070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:40.955161 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.954181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:40.955161 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.954302    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.954279931 +0000 UTC m=+32.578088139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:40.955161 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.954303    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 23:53:40.955161 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.954356    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:40.955161 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.954384    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.954372245 +0000 UTC m=+32.578180442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : secret "router-metrics-certs-default" not found
Oct 30 23:53:40.955161 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:40.954408    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:41.954392009 +0000 UTC m=+32.578200204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:40.960864 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:40.960828    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cac373_8436_4b03_a28b_0fa1bf8e7d13.slice/crio-c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a WatchSource:0}: Error finding container c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a: Status 404 returned error can't find the container with id c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a
Oct 30 23:53:40.974505 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.968277    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"]
Oct 30 23:53:40.981866 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:40.981843    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda286ab8f_3f4f_4621_8b10_3730b4e114f7.slice/crio-21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e WatchSource:0}: Error finding container 21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e: Status 404 returned error can't find the container with id 21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e
Oct 30 23:53:40.992387 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:40.992164    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"]
Oct 30 23:53:41.055430 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.055400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:41.055547 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.055457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:41.055547 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.055511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:41.055547 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.055539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:41.055689 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.055585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:41.055689 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055601    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:41.055689 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055620    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:41.055689 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055675    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:41.055689 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055601    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 23:53:41.055689 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055686    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055677    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.055658178 +0000 UTC m=+32.679466380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055677    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055688    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055762    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert podName:1b949609-5560-4b48-a5b7-229d7430d7c0 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.055747927 +0000 UTC m=+32.679556124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-k7rhq" (UID: "1b949609-5560-4b48-a5b7-229d7430d7c0") : secret "networking-console-plugin-cert" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055798    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.055780735 +0000 UTC m=+32.679588934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055824    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls podName:d1384f8a-a2ad-4e55-b6e5-14ab134fa489 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.055815292 +0000 UTC m=+32.679623513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls") pod "dns-default-shskm" (UID: "d1384f8a-a2ad-4e55-b6e5-14ab134fa489") : secret "dns-default-metrics-tls" not found
Oct 30 23:53:41.055900 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.055843    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls podName:964f2946-7176-4f03-b41f-03bd65aa8346 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:42.055832741 +0000 UTC m=+32.679640927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h2j9x" (UID: "964f2946-7176-4f03-b41f-03bd65aa8346") : secret "samples-operator-tls" not found
Oct 30 23:53:41.071702 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.071681    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"]
Oct 30 23:53:41.074228 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:41.074207    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578a9d2b_c161_4a91_8f5a_8cf5a891a2bb.slice/crio-805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4 WatchSource:0}: Error finding container 805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4: Status 404 returned error can't find the container with id 805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4
Oct 30 23:53:41.117150 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.117128    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls" event={"ID":"daf52a97-c6ca-4fba-a77b-67841614c162","Type":"ContainerStarted","Data":"8e19fb8ace6bbe6512ce9a1a0a313f1271f12b3a3686238d2f88bee2fa087955"}
Oct 30 23:53:41.118174 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.118151    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mb6hw" event={"ID":"c4cac373-8436-4b03-a28b-0fa1bf8e7d13","Type":"ContainerStarted","Data":"c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a"}
Oct 30 23:53:41.119503 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.119481    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2qk9k" event={"ID":"be5dad9f-114f-4b12-bd51-217374c94fc0","Type":"ContainerStarted","Data":"ce1eae643426b7eff012b472f5bfba95d5949478885978b37be799897b8b6d34"}
Oct 30 23:53:41.119568 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.119520    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2qk9k" event={"ID":"be5dad9f-114f-4b12-bd51-217374c94fc0","Type":"ContainerStarted","Data":"0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2"}
Oct 30 23:53:41.120790 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.120772    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kgs9w" event={"ID":"25170a0c-24d2-48b8-80fa-d2d2780a41a3","Type":"ContainerStarted","Data":"88b8ccc9e55799838297a85a23a1fdba73d17273682e0fe2351af166bbe59c9c"}
Oct 30 23:53:41.120870 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.120806    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kgs9w" event={"ID":"25170a0c-24d2-48b8-80fa-d2d2780a41a3","Type":"ContainerStarted","Data":"cf822d012c8578808fc8a406edc23586a4c395b0ac628606d0eef4b2d5ba348a"}
Oct 30 23:53:41.122699 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.122675    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-69tlv" event={"ID":"51bd9499-546b-4b4e-8350-7501c25b0ab2","Type":"ContainerStarted","Data":"a84649656d5063a1beec564f34c1b75513df70d10d3e6be5f44427d4f6cb95a5"}
Oct 30 23:53:41.122807 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.122714    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-69tlv" event={"ID":"51bd9499-546b-4b4e-8350-7501c25b0ab2","Type":"ContainerStarted","Data":"658cf2080493fcde45b96bb4d3c5b6f6fc869bb3ae4130c2e0e65ca8da4e5cc4"}
Oct 30 23:53:41.123923 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.123895    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" event={"ID":"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb","Type":"ContainerStarted","Data":"805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4"}
Oct 30 23:53:41.125067 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.125034    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv" event={"ID":"b25a3f44-1814-43a8-a01d-f98c870b7d9f","Type":"ContainerStarted","Data":"42ae59502822dbf55c1806cb0ef7cff61757499662a144052ce28b1aca20f1f3"}
Oct 30 23:53:41.126188 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.126148    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6" event={"ID":"a286ab8f-3f4f-4621-8b10-3730b4e114f7","Type":"ContainerStarted","Data":"21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e"}
Oct 30 23:53:41.149586 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.149566    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"]
Oct 30 23:53:41.153004 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:41.152979    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe484ab6_ac95_4849_acc0_37636302d300.slice/crio-c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2 WatchSource:0}: Error finding container c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2: Status 404 returned error can't find the container with id c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2
Oct 30 23:53:41.170042 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.169952    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kgs9w" podStartSLOduration=372.169937706 podStartE2EDuration="6m12.169937706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:41.16724486 +0000 UTC m=+31.791053062" watchObservedRunningTime="2026-10-30 23:53:41.169937706 +0000 UTC m=+31.793745908"
Oct 30 23:53:41.187052 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.187015    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-69tlv" podStartSLOduration=372.187004139 podStartE2EDuration="6m12.187004139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:41.185932265 +0000 UTC m=+31.809740467" watchObservedRunningTime="2026-10-30 23:53:41.187004139 +0000 UTC m=+31.810812340"
Oct 30 23:53:41.219010 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.218989    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrwcz"]
Oct 30 23:53:41.221561 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:41.221541    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb92403_d07f_4bc4_bf34_4a15adff4863.slice/crio-8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf WatchSource:0}: Error finding container 8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf: Status 404 returned error can't find the container with id 8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf
Oct 30 23:53:41.871587 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.871552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:41.871587 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.871586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:41.872347 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.871965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:41.877169 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.876081    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:53:41.877169 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.876187    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\""
Oct 30 23:53:41.877169 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.876296    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\""
Oct 30 23:53:41.877169 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.876388    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\""
Oct 30 23:53:41.877169 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.876514    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.962718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.962786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:41.962865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.962988    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.963046    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:43.963029108 +0000 UTC m=+34.586837305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.963418    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.963481    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:43.963449228 +0000 UTC m=+34.587257423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : secret "router-metrics-certs-default" not found
Oct 30 23:53:41.963584 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:41.963552    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:43.963542218 +0000 UTC m=+34.587350415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.064120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.064292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.064338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.064371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.064399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.064566    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.064624    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls podName:d1384f8a-a2ad-4e55-b6e5-14ab134fa489 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:44.06460625 +0000 UTC m=+34.688414435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls") pod "dns-default-shskm" (UID: "d1384f8a-a2ad-4e55-b6e5-14ab134fa489") : secret "dns-default-metrics-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065025    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065039    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065080    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:44.065065666 +0000 UTC m=+34.688873860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065119    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065129    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:42.065253 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065169    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:44.065148703 +0000 UTC m=+34.688956897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:42.066041 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065564    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 23:53:42.066041 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065615    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert podName:1b949609-5560-4b48-a5b7-229d7430d7c0 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:44.065599517 +0000 UTC m=+34.689407704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-k7rhq" (UID: "1b949609-5560-4b48-a5b7-229d7430d7c0") : secret "networking-console-plugin-cert" not found
Oct 30 23:53:42.066041 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065647    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 23:53:42.066041 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.065711    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls podName:964f2946-7176-4f03-b41f-03bd65aa8346 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:44.065693653 +0000 UTC m=+34.689501851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h2j9x" (UID: "964f2946-7176-4f03-b41f-03bd65aa8346") : secret "samples-operator-tls" not found
Oct 30 23:53:42.133799 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.133718    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms" event={"ID":"fe484ab6-ac95-4849-acc0-37636302d300","Type":"ContainerStarted","Data":"c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2"}
Oct 30 23:53:42.136832 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.136807    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrwcz" event={"ID":"7eb92403-d07f-4bc4-bf34-4a15adff4863","Type":"ContainerStarted","Data":"8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf"}
Oct 30 23:53:42.154569 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.154519    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2qk9k" podStartSLOduration=373.154503086 podStartE2EDuration="6m13.154503086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:42.153691231 +0000 UTC m=+32.777499434" watchObservedRunningTime="2026-10-30 23:53:42.154503086 +0000 UTC m=+32.778311289"
Oct 30 23:53:42.569163 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.569123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:42.572653 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.572626    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:42.671224 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.670437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:42.671224 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.670506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:42.671224 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.670713    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 23:53:42.671224 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:42.670788    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs podName:5f6789dc-0c9b-4507-9615-93c3ec632af2 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:14.670759349 +0000 UTC m=+65.294567544 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs") pod "network-metrics-daemon-vl6sr" (UID: "5f6789dc-0c9b-4507-9615-93c3ec632af2") : secret "metrics-daemon-secret" not found
Oct 30 23:53:42.676876 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.676830    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:42.802248 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.802210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:42.822967 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:42.822803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:43.184229 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:43.184199    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ktssl"]
Oct 30 23:53:43.184703 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:43.184263    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qs9j"]
Oct 30 23:53:43.236911 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:43.236859    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362b6ff8_9785_49c5_8704_db4c94a603d9.slice/crio-a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e WatchSource:0}: Error finding container a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e: Status 404 returned error can't find the container with id a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e
Oct 30 23:53:43.239867 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:43.239745    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73a9af6_0664_409b_a096_83a990ec98ce.slice/crio-560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372 WatchSource:0}: Error finding container 560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372: Status 404 returned error can't find the container with id 560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372
Oct 30 23:53:43.989490 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:43.989440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:43.989646 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:43.989525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:43.989646 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:43.989627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:43.989938 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:43.989918    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 23:53:43.990025 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:43.989997    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:47.989979416 +0000 UTC m=+38.613787601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : secret "router-metrics-certs-default" not found
Oct 30 23:53:43.990934 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:43.990597    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:47.990580613 +0000 UTC m=+38.614388814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:43.991062 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:43.990955    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:43.991062 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:43.991003    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:47.990988676 +0000 UTC m=+38.614796869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:44.092678 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.092639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:44.092927 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.092704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:44.092927 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.092739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:44.092927 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.092768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:44.092927 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.092811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:44.093189 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.092965    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:44.093189 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.092979    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:44.093189 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093038    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.093019208 +0000 UTC m=+38.716827416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:44.093454 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093435    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:44.093454 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093471    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:44.093569 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093519    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.093504132 +0000 UTC m=+38.717312330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:44.093615 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093586    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 23:53:44.093647 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093622    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls podName:964f2946-7176-4f03-b41f-03bd65aa8346 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.093611092 +0000 UTC m=+38.717419285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h2j9x" (UID: "964f2946-7176-4f03-b41f-03bd65aa8346") : secret "samples-operator-tls" not found
Oct 30 23:53:44.094818 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093678    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 23:53:44.094818 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert podName:1b949609-5560-4b48-a5b7-229d7430d7c0 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.093703635 +0000 UTC m=+38.717511829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-k7rhq" (UID: "1b949609-5560-4b48-a5b7-229d7430d7c0") : secret "networking-console-plugin-cert" not found
Oct 30 23:53:44.094818 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093769    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:53:44.094818 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:44.093798    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls podName:d1384f8a-a2ad-4e55-b6e5-14ab134fa489 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.093788036 +0000 UTC m=+38.717596218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls") pod "dns-default-shskm" (UID: "d1384f8a-a2ad-4e55-b6e5-14ab134fa489") : secret "dns-default-metrics-tls" not found
Oct 30 23:53:44.161043 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.160227    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv" event={"ID":"b25a3f44-1814-43a8-a01d-f98c870b7d9f","Type":"ContainerStarted","Data":"7d7b8d64da57f584d92f1117e3899f45b1897a21d771a2c90890d661a9238abb"}
Oct 30 23:53:44.163333 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.163212    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ktssl" event={"ID":"e73a9af6-0664-409b-a096-83a990ec98ce","Type":"ContainerStarted","Data":"560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372"}
Oct 30 23:53:44.168792 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.168766    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qs9j" event={"ID":"362b6ff8-9785-49c5-8704-db4c94a603d9","Type":"ContainerStarted","Data":"a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e"}
Oct 30 23:53:44.178851 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:44.178798    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv" podStartSLOduration=362.621389993 podStartE2EDuration="6m5.178782378s" totalImagesPullingTime="2.557392385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:44.177976323 +0000 UTC m=+34.801784524" watchObservedRunningTime="2026-10-30 23:53:44.178782378 +0000 UTC m=+34.802590598"
Oct 30 23:53:47.173146 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.171730    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hgtpq"]
Oct 30 23:53:47.202650 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.202623    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgtpq"]
Oct 30 23:53:47.202836 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.202672    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6" event={"ID":"a286ab8f-3f4f-4621-8b10-3730b4e114f7","Type":"ContainerStarted","Data":"4e177a19e18d57461800e58fdb65b7ccdcb0205328568ca5234a9b55a98412f9"}
Oct 30 23:53:47.202836 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.202698    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls" event={"ID":"daf52a97-c6ca-4fba-a77b-67841614c162","Type":"ContainerStarted","Data":"24de10e9b98066fd7c9cc90b0bc230416b46473adbfe9f8dfa46dd2bf2155833"}
Oct 30 23:53:47.202836 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.202726    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mb6hw" event={"ID":"c4cac373-8436-4b03-a28b-0fa1bf8e7d13","Type":"ContainerStarted","Data":"87744ac71e54d0974de45a85708c2714ec65fabecf24afc706f14d7ce2fe7836"}
Oct 30 23:53:47.202836 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.202750    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rnrwh-lb8r2"]
Oct 30 23:53:47.203683 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.203622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.207690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.207647    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:53:47.207825 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.207723    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:53:47.207897 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.207868    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g4klp\""
Oct 30 23:53:47.216480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.216181    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/0.log"
Oct 30 23:53:47.216480 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.216309    2578 generic.go:356] "Generic (PLEG): container finished" podID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerID="3ef70a8d569e07dcb3eeb10306d27aca00fbb6c41eeb933a56d862720d5480d8" exitCode=255
Oct 30 23:53:47.216670 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.216655    2578 scope.go:122] "RemoveContainer" containerID="3ef70a8d569e07dcb3eeb10306d27aca00fbb6c41eeb933a56d862720d5480d8"
Oct 30 23:53:47.217415 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.217378    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6" podStartSLOduration=362.368939646 podStartE2EDuration="6m8.217366146s" totalImagesPullingTime="5.8484265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:47.215595623 +0000 UTC m=+37.839403827" watchObservedRunningTime="2026-10-30 23:53:47.217366146 +0000 UTC m=+37.841174339"
Oct 30 23:53:47.220795 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.220774    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rnrwh-lb8r2"]
Oct 30 23:53:47.220890 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.220813    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" event={"ID":"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb","Type":"ContainerDied","Data":"3ef70a8d569e07dcb3eeb10306d27aca00fbb6c41eeb933a56d862720d5480d8"}
Oct 30 23:53:47.220950 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.220907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.231625 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.231172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.231625 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.231224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.231625 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.231319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-data-volume\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.232690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.231691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.232690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.232210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.232690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.232230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdnh\" (UniqueName: \"kubernetes.io/projected/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-api-access-wkdnh\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.232690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.232306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.232690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.232334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-crio-socket\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.232690 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.232362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.275494 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.274982    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls" podStartSLOduration=335.383538144 podStartE2EDuration="5m41.274964107s" totalImagesPullingTime="5.891425963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:47.253273966 +0000 UTC m=+37.877082170" watchObservedRunningTime="2026-10-30 23:53:47.274964107 +0000 UTC m=+37.898772310"
Oct 30 23:53:47.278036 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.276787    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mb6hw" podStartSLOduration=335.957589659 podStartE2EDuration="5m41.276775046s" totalImagesPullingTime="5.319185387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:47.274689342 +0000 UTC m=+37.898497546" watchObservedRunningTime="2026-10-30 23:53:47.276775046 +0000 UTC m=+37.900583248"
Oct 30 23:53:47.333838 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.333809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.333923 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.333863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.333957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.333918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-data-volume\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.333957 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.333946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.334075 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.333996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.334075 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdnh\" (UniqueName: \"kubernetes.io/projected/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-api-access-wkdnh\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.334171 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.334171 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-crio-socket\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.334171 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.334342 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334255    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-data-volume\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.334342 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.334331    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:47.334442 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.334382    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:47.834366212 +0000 UTC m=+38.458174422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:47.334673 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.334545    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:53:47.334673 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-crio-socket\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.334673 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.334634    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:47.834594069 +0000 UTC m=+38.458402263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : configmap "kube-rbac-proxy" not found
Oct 30 23:53:47.334925 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.334813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.335315 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.335265    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.336171 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.336106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.347438 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.347390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdnh\" (UniqueName: \"kubernetes.io/projected/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-api-access-wkdnh\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.348081 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.347969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.575595 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.575568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:47.771901 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.771866    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"]
Oct 30 23:53:47.789804 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.789401    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"]
Oct 30 23:53:47.789804 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.789534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:47.805363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.805336    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"]
Oct 30 23:53:47.805363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.805377    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"]
Oct 30 23:53:47.805631 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.805448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:47.825389 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.825350    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:47.825572 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.825547    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:53:47.846100 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.846070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zfx\" (UniqueName: \"kubernetes.io/projected/500490ba-4b0f-4e40-9f8e-3906d62c458c-kube-api-access-f5zfx\") pod \"migrator-66df5d496b-rx6l8\" (UID: \"500490ba-4b0f-4e40-9f8e-3906d62c458c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:47.846242 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.846165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.846242 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.846235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltt9r\" (UniqueName: \"kubernetes.io/projected/88aedb21-8c5c-457b-a5c1-b778c33565c3-kube-api-access-ltt9r\") pod \"migrator-66df5d496b-fjlhr\" (UID: \"88aedb21-8c5c-457b-a5c1-b778c33565c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:47.846363 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.846316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:47.846479 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.846444    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:47.846540 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.846522    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.846503993 +0000 UTC m=+39.470312195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:47.846978 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.846899    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:53:47.846978 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:47.846946    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:48.846929081 +0000 UTC m=+39.470737275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : configmap "kube-rbac-proxy" not found
Oct 30 23:53:47.859209 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.859182    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sdzl9\""
Oct 30 23:53:47.923384 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.923358    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rnrwh-lb8r2"]
Oct 30 23:53:47.925138 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:47.925088    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4498189b_b2cb_46e0_b20f_926be273f49a.slice/crio-5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94 WatchSource:0}: Error finding container 5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94: Status 404 returned error can't find the container with id 5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94
Oct 30 23:53:47.947680 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.947646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltt9r\" (UniqueName: \"kubernetes.io/projected/88aedb21-8c5c-457b-a5c1-b778c33565c3-kube-api-access-ltt9r\") pod \"migrator-66df5d496b-fjlhr\" (UID: \"88aedb21-8c5c-457b-a5c1-b778c33565c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:47.947810 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.947778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zfx\" (UniqueName: \"kubernetes.io/projected/500490ba-4b0f-4e40-9f8e-3906d62c458c-kube-api-access-f5zfx\") pod \"migrator-66df5d496b-rx6l8\" (UID: \"500490ba-4b0f-4e40-9f8e-3906d62c458c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:47.962155 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.962130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zfx\" (UniqueName: \"kubernetes.io/projected/500490ba-4b0f-4e40-9f8e-3906d62c458c-kube-api-access-f5zfx\") pod \"migrator-66df5d496b-rx6l8\" (UID: \"500490ba-4b0f-4e40-9f8e-3906d62c458c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:47.964374 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:47.964349    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltt9r\" (UniqueName: \"kubernetes.io/projected/88aedb21-8c5c-457b-a5c1-b778c33565c3-kube-api-access-ltt9r\") pod \"migrator-66df5d496b-fjlhr\" (UID: \"88aedb21-8c5c-457b-a5c1-b778c33565c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:48.049125 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.049084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:48.049278 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.049209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:48.049278 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.049252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:48.049456 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.049442    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.049424958 +0000 UTC m=+46.673233142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:48.049580 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.049565    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:48.049639 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.049617    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.049601725 +0000 UTC m=+46.673409921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:48.049700 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.049673    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 23:53:48.049751 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.049709    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.049698227 +0000 UTC m=+46.673506425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : secret "router-metrics-certs-default" not found
Oct 30 23:53:48.118539 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.118299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:48.142943 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.142913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:48.143446 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.143423    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-84r2z"]
Oct 30 23:53:48.150503 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.150397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:48.150602 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.150587    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:48.150677 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.150607    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:48.150677 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.150660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.150642782 +0000 UTC m=+46.774450962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:48.150833 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.150451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:48.152834 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.152812    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 23:53:48.152937 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.152874    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls podName:964f2946-7176-4f03-b41f-03bd65aa8346 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.152858099 +0000 UTC m=+46.776666295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h2j9x" (UID: "964f2946-7176-4f03-b41f-03bd65aa8346") : secret "samples-operator-tls" not found
Oct 30 23:53:48.152937 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.152910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:48.153055 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.152944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:48.153055 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.153001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:48.153221 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153190    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:48.153221 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153203    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:48.158201 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153240    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.153227308 +0000 UTC m=+46.777035496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:48.158201 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153300    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 23:53:48.158201 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153335    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert podName:1b949609-5560-4b48-a5b7-229d7430d7c0 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.153324088 +0000 UTC m=+46.777132287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-k7rhq" (UID: "1b949609-5560-4b48-a5b7-229d7430d7c0") : secret "networking-console-plugin-cert" not found
Oct 30 23:53:48.158201 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153403    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:53:48.158201 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.153433    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls podName:d1384f8a-a2ad-4e55-b6e5-14ab134fa489 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.153423025 +0000 UTC m=+46.777231221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls") pod "dns-default-shskm" (UID: "d1384f8a-a2ad-4e55-b6e5-14ab134fa489") : secret "dns-default-metrics-tls" not found
Oct 30 23:53:48.168498 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.168451    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-84r2z"]
Oct 30 23:53:48.168602 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.168585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.174887 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.174339    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:53:48.174887 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.174454    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pr9l6\""
Oct 30 23:53:48.174887 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.174696    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:53:48.174887 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.174794    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:53:48.174887 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.174845    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:53:48.228070 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.227946    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:53:48.228927 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.228908    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/0.log"
Oct 30 23:53:48.229031 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.228955    2578 generic.go:356] "Generic (PLEG): container finished" podID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160" exitCode=255
Oct 30 23:53:48.229489 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.229254    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" event={"ID":"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb","Type":"ContainerDied","Data":"887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160"}
Oct 30 23:53:48.229489 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.229314    2578 scope.go:122] "RemoveContainer" containerID="3ef70a8d569e07dcb3eeb10306d27aca00fbb6c41eeb933a56d862720d5480d8"
Oct 30 23:53:48.229895 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.229875    2578 scope.go:122] "RemoveContainer" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160"
Oct 30 23:53:48.230113 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.230096    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zzxg7_openshift-console-operator(578a9d2b-c161-4a91-8f5a-8cf5a891a2bb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb"
Oct 30 23:53:48.236075 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.235648    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2" event={"ID":"4498189b-b2cb-46e0-b20f-926be273f49a","Type":"ContainerStarted","Data":"b2207b815352754bc0a3c1df853d799b6a2f40fe5577eff1e8e70d4f817d4408"}
Oct 30 23:53:48.236075 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.235695    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2" event={"ID":"4498189b-b2cb-46e0-b20f-926be273f49a","Type":"ContainerStarted","Data":"5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94"}
Oct 30 23:53:48.242097 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.241943    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrwcz" event={"ID":"7eb92403-d07f-4bc4-bf34-4a15adff4863","Type":"ContainerStarted","Data":"274eb8a0af60296bf608b4bded105aa6d96284bb686fe47abce84928ecff9f9d"}
Oct 30 23:53:48.257510 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.254716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72c64997-e326-411c-9f16-8ad1671ef849-signing-key\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.257510 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.254957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86sq\" (UniqueName: \"kubernetes.io/projected/72c64997-e326-411c-9f16-8ad1671ef849-kube-api-access-q86sq\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.257510 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.255119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72c64997-e326-411c-9f16-8ad1671ef849-signing-cabundle\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.265917 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.265862    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2" podStartSLOduration=1.2658443099999999 podStartE2EDuration="1.26584431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:48.264382572 +0000 UTC m=+38.888190779" watchObservedRunningTime="2026-10-30 23:53:48.26584431 +0000 UTC m=+38.889652513"
Oct 30 23:53:48.280799 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.280757    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mrwcz" podStartSLOduration=1.970030619 podStartE2EDuration="8.28074555s" totalImagesPullingTime="6.310714931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:48.279971262 +0000 UTC m=+38.903779466" watchObservedRunningTime="2026-10-30 23:53:48.28074555 +0000 UTC m=+38.904553752"
Oct 30 23:53:48.356530 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.356494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72c64997-e326-411c-9f16-8ad1671ef849-signing-cabundle\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.356693 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.356626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72c64997-e326-411c-9f16-8ad1671ef849-signing-key\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.356768 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.356755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86sq\" (UniqueName: \"kubernetes.io/projected/72c64997-e326-411c-9f16-8ad1671ef849-kube-api-access-q86sq\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.357275 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.357247    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72c64997-e326-411c-9f16-8ad1671ef849-signing-cabundle\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.364613 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.364555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72c64997-e326-411c-9f16-8ad1671ef849-signing-key\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.368711 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.368669    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86sq\" (UniqueName: \"kubernetes.io/projected/72c64997-e326-411c-9f16-8ad1671ef849-kube-api-access-q86sq\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.424322 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.424292    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"]
Oct 30 23:53:48.427454 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:48.427430    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aedb21_8c5c_457b_a5c1_b778c33565c3.slice/crio-2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731 WatchSource:0}: Error finding container 2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731: Status 404 returned error can't find the container with id 2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731
Oct 30 23:53:48.467827 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.467800    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"]
Oct 30 23:53:48.470351 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:48.470325    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500490ba_4b0f_4e40_9f8e_3906d62c458c.slice/crio-3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a WatchSource:0}: Error finding container 3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a: Status 404 returned error can't find the container with id 3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a
Oct 30 23:53:48.491868 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.491841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:48.792232 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.792157    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-84r2z"]
Oct 30 23:53:48.796606 ip-10-0-143-16 kubenswrapper[2578]: W1030 23:53:48.796579    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c64997_e326_411c_9f16_8ad1671ef849.slice/crio-197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d WatchSource:0}: Error finding container 197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d: Status 404 returned error can't find the container with id 197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d
Oct 30 23:53:48.862504 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.862456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:48.862636 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.862617    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:53:48.862699 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.862690    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:50.862669412 +0000 UTC m=+41.486477609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : configmap "kube-rbac-proxy" not found
Oct 30 23:53:48.862822 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:48.862801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:48.862953 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.862936    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:48.863007 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:48.863001    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:50.862984979 +0000 UTC m=+41.486793179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:49.249611 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.248376    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:53:49.249611 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.248953    2578 scope.go:122] "RemoveContainer" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160"
Oct 30 23:53:49.249611 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:49.249159    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zzxg7_openshift-console-operator(578a9d2b-c161-4a91-8f5a-8cf5a891a2bb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb"
Oct 30 23:53:49.252426 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.252329    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr" event={"ID":"88aedb21-8c5c-457b-a5c1-b778c33565c3","Type":"ContainerStarted","Data":"2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731"}
Oct 30 23:53:49.254016 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.253990    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8" event={"ID":"500490ba-4b0f-4e40-9f8e-3906d62c458c","Type":"ContainerStarted","Data":"3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a"}
Oct 30 23:53:49.255769 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.255728    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms" event={"ID":"fe484ab6-ac95-4849-acc0-37636302d300","Type":"ContainerStarted","Data":"974f8e566f22b7323c4c0c09b41c4ae0e30f9043f116045f5732d8d4c069f4a5"}
Oct 30 23:53:49.259493 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.259086    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z" event={"ID":"72c64997-e326-411c-9f16-8ad1671ef849","Type":"ContainerStarted","Data":"2176406618b23e67b054f167b96cd4878f13e8802981bb4f28f2f290aff7c8e3"}
Oct 30 23:53:49.259493 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.259123    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z" event={"ID":"72c64997-e326-411c-9f16-8ad1671ef849","Type":"ContainerStarted","Data":"197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d"}
Oct 30 23:53:49.352630 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:49.350883    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z" podStartSLOduration=1.35086354 podStartE2EDuration="1.35086354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:49.301020588 +0000 UTC m=+39.924828791" watchObservedRunningTime="2026-10-30 23:53:49.35086354 +0000 UTC m=+39.974671745"
Oct 30 23:53:50.266249 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.266214    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qs9j" event={"ID":"362b6ff8-9785-49c5-8704-db4c94a603d9","Type":"ContainerStarted","Data":"12226c1f975f83c9d9a410ae27ef1b03e606c5f720287cc1c333c7cc6eb0db71"}
Oct 30 23:53:50.266744 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.266711    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:50.295181 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.295123    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms" podStartSLOduration=337.058605969 podStartE2EDuration="5m44.295103568s" totalImagesPullingTime="7.236497599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:49.351825855 +0000 UTC m=+39.975634078" watchObservedRunningTime="2026-10-30 23:53:50.295103568 +0000 UTC m=+40.918911771"
Oct 30 23:53:50.535705 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.535631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ccfzb_3bbf2354-7b35-4374-adc3-e694b71f6e9e/dns-node-resolver/0.log"
Oct 30 23:53:50.784693 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.784662    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:50.784849 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.784731    2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:50.785125 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.785107    2578 scope.go:122] "RemoveContainer" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160"
Oct 30 23:53:50.785337 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:50.785313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zzxg7_openshift-console-operator(578a9d2b-c161-4a91-8f5a-8cf5a891a2bb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb"
Oct 30 23:53:50.892433 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.892395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:50.892611 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:50.892539    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:50.892611 ip-10-0-143-16 kubenswrapper[2578]: E1030 23:53:50.892600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:54.892583154 +0000 UTC m=+45.516391355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:50.892749 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.892648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:50.893293 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:50.893264    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:51.270306 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:51.270257    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr" event={"ID":"88aedb21-8c5c-457b-a5c1-b778c33565c3","Type":"ContainerStarted","Data":"73e03d3f4efbc9bb48154660d565906ac91a0e928cecc5649ca596aed03cb430"}
Oct 30 23:53:51.272221 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:51.272187    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8" event={"ID":"500490ba-4b0f-4e40-9f8e-3906d62c458c","Type":"ContainerStarted","Data":"5f03644fd2027f9943d6862807169be492b90e38bb255b7b3162f363ec9bd72e"}
Oct 30 23:53:51.287823 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:51.287772    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2qs9j" podStartSLOduration=35.016480293 podStartE2EDuration="41.287757444s" totalImagesPullingTime="6.271277151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:50.295553865 +0000 UTC m=+40.919362070" watchObservedRunningTime="2026-10-30 23:53:51.287757444 +0000 UTC m=+41.911565649"
Oct 30 23:53:51.288126 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:51.288096    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr" podStartSLOduration=1.90950615 podStartE2EDuration="4.288086632s" totalImagesPullingTime="2.378580482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:51.287523186 +0000 UTC m=+41.911331389" watchObservedRunningTime="2026-10-30 23:53:51.288086632 +0000 UTC m=+41.911894835"
Oct 30 23:53:51.305346 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:51.305303    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8" podStartSLOduration=1.972948503 podStartE2EDuration="4.305288687s" totalImagesPullingTime="2.332340184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:51.303773562 +0000 UTC m=+41.927581763" watchObservedRunningTime="2026-10-30 23:53:51.305288687 +0000 UTC m=+41.929096889"
Oct 30 23:53:51.746634 ip-10-0-143-16 kubenswrapper[2578]: I1030 23:53:51.746614    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bndnt_6e24b7db-f2a1-4500-9d82-963292cb7765/node-ca/0.log"
Oct 30 23:53:51.991256 ip-10-0-143-16 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 23:53:52.012922 ip-10-0-143-16 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 23:53:52.013098 ip-10-0-143-16 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 23:53:52.013148 ip-10-0-143-16 systemd[1]: kubelet.service: Consumed 3.154s CPU time.
Oct 30 23:53:52.022132 ip-10-0-143-16 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:53:52.102606 ip-10-0-143-16 kubenswrapper[5078]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:53:52.102606 ip-10-0-143-16 kubenswrapper[5078]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:53:52.102606 ip-10-0-143-16 kubenswrapper[5078]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:53:52.102606 ip-10-0-143-16 kubenswrapper[5078]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:53:52.104658 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104581    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.104658 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104657    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.104658 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104660    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.104658 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104663    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104666    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104669    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104672    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104675    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104678    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104681    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104683    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104686    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104688    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104698    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104701    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104703    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104706    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104708    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104711    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104713    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104716    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104718    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.104789 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104721    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104723    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104726    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104728    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104731    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104734    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104736    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104739    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104741    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104743    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104746    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104749    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104751    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104754    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104757    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104760    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104762    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104765    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104767    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104770    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.105245 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104772    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104774    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104777    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104779    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104781    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104784    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104786    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104788    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104791    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104794    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104797    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104799    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104802    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104804    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104807    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104809    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104815    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104819    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104824    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.105744 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104828    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104831    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104835    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104838    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104841    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104844    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104847    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104849    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104852    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104854    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104856    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104860    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104862    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104865    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104867    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104870    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104872    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104875    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104878    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.106221 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104880    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104882    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104885    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104887    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104890    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104892    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104894    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104897    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104899    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104901    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104905    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104909    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104912    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104915    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104918    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104920    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104923    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104926    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104928    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.106709 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104931    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104933    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104937    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104940    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104942    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104945    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.104947    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105707    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105714    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105716    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105719    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105722    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105724    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105727    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105729    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105732    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105734    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105737    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105740    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105742    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.107159 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105745    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105747    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105750    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105755    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105758    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105760    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105763    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105765    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105767    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105770    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105773    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105775    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105778    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105781    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105783    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105786    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105788    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105791    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105793    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105796    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.107674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105799    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105802    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105804    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105807    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105809    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105812    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105814    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105817    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105819    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105822    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105825    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105829    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105831    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105834    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105837    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105840    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105843    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105845    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105849    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.108194 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105852    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105854    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105856    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105859    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105862    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105864    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105867    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105869    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105871    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105880    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105883    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105886    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105888    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105891    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105894    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105896    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105899    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105901    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105904    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105906    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.108674 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105909    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105911    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105914    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105916    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105919    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105921    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105923    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105926    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105930    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105932    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105935    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105938    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105940    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105943    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105945    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105948    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105950    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105954    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105958    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.109190 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105969    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105973    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105976    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105978    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105981    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105983    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105986    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105989    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105993    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105995    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.105998    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106000    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106003    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106005    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106008    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106496    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106502    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106504    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106507    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.109655 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106510    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106512    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106516    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106519    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106522    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106524    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106527    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106529    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106532    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106535    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106537    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106540    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106542    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106544    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106548    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106550    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106553    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106555    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106557    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.110126 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106560    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106566    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106571    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106574    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106578    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106581    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106583    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106586    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106589    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106591    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106594    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106596    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106599    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106601    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106603    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106607    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106610    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106613    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106616    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.110751 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106618    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106621    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106624    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106626    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106629    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106633    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106636    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106639    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106641    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106643    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106646    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106648    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106651    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106653    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106656    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106659    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106661    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106664    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106666    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106669    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.111569 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106671    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106674    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106676    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106679    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106681    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106683    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106686    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106688    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106692    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106695    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106699    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106702    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106705    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106708    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106710    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106713    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106716    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106718    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106721    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.112421 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106723    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106726    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106729    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106732    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106735    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106737    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106739    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106742    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106744    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106749    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106752    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106755    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106757    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106759    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106762    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106764    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106767    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106769    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106772    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.113240 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106774    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106776    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106780    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106784    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106787    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.106789    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106900    5078 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106918    5078 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106924    5078 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106928    5078 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106932    5078 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106935    5078 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106939    5078 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106943    5078 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106946    5078 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106949    5078 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106953    5078 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106956    5078 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106959    5078 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106962    5078 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106965    5078 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106968    5078 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106970    5078 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:53:52.114076 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106981    5078 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106988    5078 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106991    5078 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106993    5078 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.106996    5078 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107000    5078 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107004    5078 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107006    5078 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107009    5078 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107012    5078 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107015    5078 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107020    5078 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107024    5078 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107027    5078 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107030    5078 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107034    5078 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107037    5078 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107040    5078 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107042    5078 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107045    5078 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107048    5078 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107055    5078 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107058    5078 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107061    5078 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:53:52.114905 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107064    5078 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107067    5078 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107071    5078 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107074    5078 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107076    5078 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107079    5078 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107082    5078 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107085    5078 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107088    5078 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107096    5078 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107099    5078 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107102    5078 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107105    5078 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107109    5078 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107112    5078 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107115    5078 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107118    5078 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107121    5078 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107124    5078 flags.go:64] FLAG: --help="false"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107128    5078 flags.go:64] FLAG: --hostname-override="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107132    5078 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107135    5078 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107138    5078 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:53:52.115528 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107141    5078 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107145    5078 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107148    5078 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107151    5078 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107153    5078 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107156    5078 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107159    5078 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107162    5078 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107165    5078 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107168    5078 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107171    5078 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107174    5078 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107177    5078 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107180    5078 flags.go:64] FLAG: --lock-file=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107182    5078 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107185    5078 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107188    5078 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107194    5078 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107196    5078 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107202    5078 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107205    5078 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107208    5078 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107211    5078 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107214    5078 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107217    5078 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:53:52.116136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107223    5078 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107227    5078 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107242    5078 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107249    5078 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107254    5078 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107257    5078 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107260    5078 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107263    5078 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107265    5078 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107268    5078 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107275    5078 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107278    5078 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107281    5078 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107284    5078 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107287    5078 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107290    5078 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107292    5078 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107295    5078 flags.go:64] FLAG: --port="10250"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107298    5078 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107301    5078 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0111924f3c1f82f3f"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107304    5078 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107306    5078 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107309    5078 flags.go:64] FLAG: --register-node="true"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107312    5078 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107316    5078 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107318    5078 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:53:52.116748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107321    5078 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107324    5078 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107328    5078 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107331    5078 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107334    5078 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107336    5078 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107339    5078 flags.go:64] FLAG: --runonce="false"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107342    5078 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107345    5078 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107349    5078 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107353    5078 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107356    5078 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107359    5078 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107361    5078 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107364    5078 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107367    5078 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107370    5078 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107373    5078 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107375    5078 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107378    5078 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107381    5078 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107384    5078 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107389    5078 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107392    5078 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107394    5078 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:53:52.117450 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107399    5078 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107401    5078 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107404    5078 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107407    5078 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107409    5078 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107412    5078 flags.go:64] FLAG: --v="2"
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107416    5078 flags.go:64] FLAG: --version="false"
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107420    5078 flags.go:64] FLAG: --vmodule=""
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107424    5078 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107428    5078 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107619    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107625    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107628    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107631    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107633    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107636    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107641    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107645    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107648    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107651    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107653    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.118135 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107656    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107658    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107661    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107663    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107666    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107669    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107671    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107674    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107677    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107679    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107682    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107685    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107688    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107690    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107693    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107696    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107698    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107701    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107703    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107705    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.118685 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107708    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107711    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107713    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107716    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107718    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107721    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107723    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107726    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107730    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107732    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107735    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107739    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107742    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107746    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107749    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107751    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107755    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107759    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107762    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.119174 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107765    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107767    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107770    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107772    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107775    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107777    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107780    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107782    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107784    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107787    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107789    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107792    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107794    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107796    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107799    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107804    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107806    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107809    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107811    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107813    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.119752 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107816    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107820    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107823    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107825    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107827    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107830    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107833    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107835    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107838    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107840    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107843    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107845    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107848    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107850    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107853    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107855    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107858    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107860    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107863    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107865    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.120442 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107868    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107871    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107873    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107875    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107878    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107880    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107882    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107885    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107893    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107896    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107898    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107900    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107903    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107907    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107910    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107912    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.121333 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.107940    5078 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107974    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107979    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107983    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107986    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107988    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107992    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.107996    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108000    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108002    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108005    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108007    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108010    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108012    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108015    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108017    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108019    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108022    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108024    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.122100 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108027    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108030    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108032    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108034    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108037    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108039    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108042    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108045    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108047    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108049    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108053    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108055    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108058    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108060    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108062    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108065    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108068    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108070    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108072    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108075    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.122670 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108077    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108079    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108082    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108084    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108087    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108089    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108092    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108094    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108097    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108099    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108101    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108105    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108108    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108110    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108113    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108115    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108119    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108122    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108124    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108127    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.123313 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108130    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108132    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108135    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108138    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108140    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108143    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108145    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108147    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108150    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108152    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108155    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108157    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108160    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108163    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108165    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108167    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108170    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108172    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108174    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.123831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108177    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108179    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108181    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108184    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108187    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108190    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108193    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108195    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108198    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108200    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108203    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108206    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108210    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108213    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108216    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108219    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108221    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108224    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108226    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108228    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.124306 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108231    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108233    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108236    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108238    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108240    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108243    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108245    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108247    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.108250    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.112910    5078 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.112928    5078 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.112953    5078 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.112960    5078 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113043    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113050    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113057    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113064    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113069    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113073    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.124802 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113077    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113082    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113086    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113091    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113095    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113099    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113103    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113108    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113112    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113116    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113121    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113125    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113130    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113134    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113138    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113143    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113147    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113152    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113156    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113160    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.125282 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113165    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113169    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113173    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113178    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113183    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113187    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113191    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113195    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113200    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113204    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113209    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113213    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113217    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113222    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113226    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113230    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113234    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113238    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113242    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113246    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.125870 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113249    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113254    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113258    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113262    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113266    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113271    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113275    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113279    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113286    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113291    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113295    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113299    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113303    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113307    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113311    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113316    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113321    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113326    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113330    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113334    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.126370 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113339    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113344    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113348    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113353    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113357    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113361    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113365    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113369    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113373    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113377    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113382    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113386    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113390    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113394    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113398    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113402    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113406    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113411    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113415    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.126877 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113419    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113423    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113428    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113432    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113436    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113440    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113444    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113448    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113452    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113456    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113485    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113489    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113494    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113498    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113505    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113511    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113515    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113519    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113523    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.127329 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113527    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.113532    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.113573    5078 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114212    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114230    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114235    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114243    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114270    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114276    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114287    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114292    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114297    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114301    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114305    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114310    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.127811 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114315    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114319    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114324    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114328    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114333    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114337    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114342    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114351    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114355    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114360    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114364    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114368    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114372    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114376    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114380    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114385    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114389    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114393    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114398    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.128170 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114408    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114412    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114418    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114422    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114426    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114430    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114434    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114439    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114443    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114447    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114452    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114456    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114484    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114495    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114499    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114503    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114507    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114512    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114516    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114520    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.128747 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114525    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114530    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114537    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114544    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114549    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114559    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114563    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114568    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114572    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114576    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114584    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114588    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114593    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114597    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114601    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114606    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114610    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114614    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114623    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.129290 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114627    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114631    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114635    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114639    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114643    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114648    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114652    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114656    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114660    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114665    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.114715    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115255    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115264    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115268    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115273    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115277    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115281    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115285    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115289    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.129821 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115293    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115298    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115302    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115306    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115310    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115317    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115322    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115326    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115331    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115335    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115340    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115344    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115349    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115353    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115357    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115361    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115365    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.130288 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.115416    5078 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115491    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115500    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115505    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115509    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115511    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115514    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115517    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115519    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115522    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115525    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115527    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115530    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115533    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115537    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115540    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115543    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115545    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115548    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:53:52.130760 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115550    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115553    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115556    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115558    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115561    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115564    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115566    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115569    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115571    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115574    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115577    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115579    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115582    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115585    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115587    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115590    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115592    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115594    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115597    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:53:52.131215 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115599    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115602    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115604    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115607    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115610    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115612    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115615    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115617    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115620    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115622    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115625    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115628    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115633    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115638    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115641    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115644    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115647    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115650    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115654    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:53:52.131746 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115657    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115659    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115662    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115665    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115668    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115670    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115673    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115675    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115678    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115681    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115683    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115685    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115688    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115690    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115693    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115695    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115698    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115700    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115703    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115707    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:53:52.132228 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115710    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115712    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115715    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115717    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115720    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115722    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115725    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115727    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115730    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115733    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115736    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115748    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115751    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115754    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115756    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115759    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115761    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115764    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115766    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115769    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:53:52.132772 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115772    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115774    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115777    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115779    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115781    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115784    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115787    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115789    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115791    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:52.115796    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.115854    5078 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.116802    5078 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.116869    5078 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.117061    5078 server.go:1014] "Starting client certificate rotation"
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.117212    5078 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.117281    5078 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:48:09 +0000 UTC" deadline="2028-07-25 00:20:38.979801488 +0000 UTC"
Oct 30 23:53:52.133250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.117321    5078 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15192h26m46.862483851s"
Oct 30 23:53:52.133666 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.117579    5078 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:53:52.133666 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.118649    5078 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:53:52.133666 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.119880    5078 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:53:52.133666 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.122020    5078 log.go:25] "Validated CRI v1 image API"
Oct 30 23:53:52.133666 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.122825    5078 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:53:52.133666 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.124788    5078 fs.go:135] Filesystem UUIDs: map[4611be2a-145d-40c5-b010-c0f6aa0e0aa4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f1a614a9-ed6c-476c-a5bb-0a9c4b1703a5:/dev/nvme0n1p3]
Oct 30 23:53:52.133842 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.124802    5078 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42ae59502822dbf55c1806cb0ef7cff61757499662a144052ce28b1aca20f1f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42ae59502822dbf55c1806cb0ef7cff61757499662a144052ce28b1aca20f1f3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/658cf2080493fcde45b96bb4d3c5b6f6fc869bb3ae4130c2e0e65ca8da4e5cc4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/658cf2080493fcde45b96bb4d3c5b6f6fc869bb3ae4130c2e0e65ca8da4e5cc4/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e19fb8ace6bbe6512ce9a1a0a313f1271f12b3a3686238d2f88bee2fa087955/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e19fb8ace6bbe6512ce9a1a0a313f1271f12b3a3686238d2f88bee2fa087955/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf822d012c8578808fc8a406edc23586a4c395b0ac628606d0eef4b2d5ba348a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf822d012c8578808fc8a406edc23586a4c395b0ac628606d0eef4b2d5ba348a/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volumes/kubernetes.io~projected/kube-api-access-k2rft:{mountpoint:/var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volumes/kubernetes.io~projected/kube-api-access-k2rft major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25170a0c-24d2-48b8-80fa-d2d2780a41a3/volumes/kubernetes.io~projected/kube-api-access-xkwq4:{mountpoint:/var/lib/kubelet/pods/25170a0c-24d2-48b8-80fa-d2d2780a41a3/volumes/kubernetes.io~projected/kube-api-access-xkwq4 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/362b6ff8-9785-49c5-8704-db4c94a603d9/volumes/kubernetes.io~projected/kube-api-access-mhm77:{mountpoint:/var/lib/kubelet/pods/362b6ff8-9785-49c5-8704-db4c94a603d9/volumes/kubernetes.io~projected/kube-api-access-mhm77 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~projected/kube-api-access-npz6x:{mountpoint:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~projected/kube-api-access-npz6x major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bbf2354-7b35-4374-adc3-e694b71f6e9e/volumes/kubernetes.io~projected/kube-api-access-sgh87:{mountpoint:/var/lib/kubelet/pods/3bbf2354-7b35-4374-adc3-e694b71f6e9e/volumes/kubernetes.io~projected/kube-api-access-sgh87 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4498189b-b2cb-46e0-b20f-926be273f49a/volumes/kubernetes.io~projected/kube-api-access-zb26f:{mountpoint:/var/lib/kubelet/pods/4498189b-b2cb-46e0-b20f-926be273f49a/volumes/kubernetes.io~projected/kube-api-access-zb26f major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/500490ba-4b0f-4e40-9f8e-3906d62c458c/volumes/kubernetes.io~projected/kube-api-access-f5zfx:{mountpoint:/var/lib/kubelet/pods/500490ba-4b0f-4e40-9f8e-3906d62c458c/volumes/kubernetes.io~projected/kube-api-access-f5zfx major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51bd9499-546b-4b4e-8350-7501c25b0ab2/volumes/kubernetes.io~projected/kube-api-access-qwggr:{mountpoint:/var/lib/kubelet/pods/51bd9499-546b-4b4e-8350-7501c25b0ab2/volumes/kubernetes.io~projected/kube-api-access-qwggr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/567d58b9-e17a-41fd-a5e2-6ec9ff24597d/volumes/kubernetes.io~projected/kube-api-access-wkdnh:{mountpoint:/var/lib/kubelet/pods/567d58b9-e17a-41fd-a5e2-6ec9ff24597d/volumes/kubernetes.io~projected/kube-api-access-wkdnh major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/volumes/kubernetes.io~projected/kube-api-access-v8tst:{mountpoint:/var/lib/kubelet/pods/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/volumes/kubernetes.io~projected/kube-api-access-v8tst major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/volumes/kubernetes.io~secret/serving-cert major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6789dc-0c9b-4507-9615-93c3ec632af2/volumes/kubernetes.io~projected/kube-api-access-t4qs8:{mountpoint:/var/lib/kubelet/pods/5f6789dc-0c9b-4507-9615-93c3ec632af2/volumes/kubernetes.io~projected/kube-api-access-t4qs8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b213ff9-3e00-426a-901f-e5ca105d37e8/volumes/kubernetes.io~projected/kube-api-access-pqdzc:{mountpoint:/var/lib/kubelet/pods/6b213ff9-3e00-426a-901f-e5ca105d37e8/volumes/kubernetes.io~projected/kube-api-access-pqdzc major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e24b7db-f2a1-4500-9d82-963292cb7765/volumes/kubernetes.io~projected/kube-api-access-4jpsf:{mountpoint:/var/lib/kubelet/pods/6e24b7db-f2a1-4500-9d82-963292cb7765/volumes/kubernetes.io~projected/kube-api-access-4jpsf major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~projected/kube-api-access-8vrjb:{mountpoint:/var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~projected/kube-api-access-8vrjb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72c64997-e326-411c-9f16-8ad1671ef849/volumes/kubernetes.io~projected/kube-api-access-q86sq:{mountpoint:/var/lib/kubelet/pods/72c64997-e326-411c-9f16-8ad1671ef849/volumes/kubernetes.io~projected/kube-api-access-q86sq major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72c64997-e326-411c-9f16-8ad1671ef849/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/72c64997-e326-411c-9f16-8ad1671ef849/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eb92403-d07f-4bc4-bf34-4a15adff4863/volumes/kubernetes.io~projected/kube-api-access-fvx2l:{mountpoint:/var/lib/kubelet/pods/7eb92403-d07f-4bc4-bf34-4a15adff4863/volumes/kubernetes.io~projected/kube-api-access-fvx2l major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eb92403-d07f-4bc4-bf34-4a15adff4863/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7eb92403-d07f-4bc4-bf34-4a15adff4863/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88aedb21-8c5c-457b-a5c1-b778c33565c3/volumes/kubernetes.io~projected/kube-api-access-ltt9r:{mountpoint:/var/lib/kubelet/pods/88aedb21-8c5c-457b-a5c1-b778c33565c3/volumes/kubernetes.io~projected/kube-api-access-ltt9r major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/964f2946-7176-4f03-b41f-03bd65aa8346/volumes/kubernetes.io~projected/kube-api-access-pqjsr:{mountpoint:/var/lib/kubelet/pods/964f2946-7176-4f03-b41f-03bd65aa8346/volumes/kubernetes.io~projected/kube-api-access-pqjsr major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a286ab8f-3f4f-4621-8b10-3730b4e114f7/volumes/kubernetes.io~projected/kube-api-access-dhmm2:{mountpoint:/var/lib/kubelet/pods/a286ab8f-3f4f-4621-8b10-3730b4e114f7/volumes/kubernetes.io~projected/kube-api-access-dhmm2 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a286ab8f-3f4f-4621-8b10-3730b4e114f7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a286ab8f-3f4f-4621-8b10-3730b4e114f7/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~projected/kube-api-access-ng52f:{mountpoint:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~projected/kube-api-access-ng52f major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7/volumes/kubernetes.io~projected/kube-api-access-fgpwz:{mountpoint:/var/lib/kubelet/pods/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7/volumes/kubernetes.io~projected/kube-api-access-fgpwz major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25a3f44-1814-43a8-a01d-f98c870b7d9f/volumes/kubernetes.io~projected/kube-api-access-tbp2v:{mountpoint:/var/lib/kubelet/pods/b25a3f44-1814-43a8-a01d-f98c870b7d9f/volumes/kubernetes.io~projected/kube-api-access-tbp2v major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be5dad9f-114f-4b12-bd51-217374c94fc0/volumes/kubernetes.io~projected/kube-api-access-nt4bc:{mountpoint:/var/lib/kubelet/pods/be5dad9f-114f-4b12-bd51-217374c94fc0/volumes/kubernetes.io~projected/kube-api-access-nt4bc major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4cac373-8436-4b03-a28b-0fa1bf8e7d13/volumes/kubernetes.io~projected/kube-api-access-fl8h7:{mountpoint:/var/lib/kubelet/pods/c4cac373-8436-4b03-a28b-0fa1bf8e7d13/volumes/kubernetes.io~projected/kube-api-access-fl8h7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4cac373-8436-4b03-a28b-0fa1bf8e7d13/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c4cac373-8436-4b03-a28b-0fa1bf8e7d13/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1090e22-f04c-4bda-b340-0966209254af/volumes/kubernetes.io~projected/kube-api-access-h2dmv:{mountpoint:/var/lib/kubelet/pods/d1090e22-f04c-4bda-b340-0966209254af/volumes/kubernetes.io~projected/kube-api-access-h2dmv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1384f8a-a2ad-4e55-b6e5-14ab134fa489/volumes/kubernetes.io~projected/kube-api-access-95p8n:{mountpoint:/var/lib/kubelet/pods/d1384f8a-a2ad-4e55-b6e5-14ab134fa489/volumes/kubernetes.io~projected/kube-api-access-95p8n major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~projected/kube-api-access-g8586:{mountpoint:/var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~projected/kube-api-access-g8586 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf52a97-c6ca-4fba-a77b-67841614c162/volumes/kubernetes.io~projected/kube-api-access-x6w5v:{mountpoint:/var/lib/kubelet/pods/daf52a97-c6ca-4fba-a77b-67841614c162/volumes/kubernetes.io~projected/kube-api-access-x6w5v major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf52a97-c6ca-4fba-a77b-67841614c162/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/daf52a97-c6ca-4fba-a77b-67841614c162/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58/volumes/kubernetes.io~projected/kube-api-access-859wz:{mountpoint:/var/lib/kubelet/pods/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58/volumes/kubernetes.io~projected/kube-api-access-859wz major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e73a9af6-0664-409b-a096-83a990ec98ce/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e73a9af6-0664-409b-a096-83a990ec98ce/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa85bf8b-61bf-4bf1-b820-05215a7eadf3/volumes/kubernetes.io~projected/kube-api-access-cb8kf:{mountpoint:/var/lib/kubelet/pods/fa85bf8b-61bf-4bf1-b820-05215a7eadf3/volumes/kubernetes.io~projected/kube-api-access-cb8kf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe484ab6-ac95-4849-acc0-37636302d300/volumes/kubernetes.io~projected/kube-api-access-r4557:{mountpoint:/var/lib/kubelet/pods/fe484ab6-ac95-4849-acc0-37636302d300/volumes/kubernetes.io~projected/kube-api-access-r4557 major:0 minor:235 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f766a90f0fb6bf86f26b5ee95dd4e2a785cc831d016bfc5f372167cd579d5b60/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8ee022a0e788875639fa1f49ab5cb442a2edbd7c5ec1f76e0e8cff1530f3d089/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c8a35f60807def4a0f568051dd0a959db5617efb0f0a351a9d4b5da82f703773/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/d24052ff235a855d9503124a3ce931cb4f77f0632d962470feecdb32b0abc75a/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3b4b79edd747d1ae8d728459f7b2efdb1309222a054149e52cdbdda2352ab9e6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f3eb9c302fa7f804c826911f7f2cb11d5ee886530ce61370d6048e1173b86559/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9d038ff01612eb8c5327909911f07ffe21e30f2ff3d0100b2462bf7e25630ccf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/30742caa280ac316878d7010853251516f108e1bb63822617442be5c222d97d9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6fc53edb3aba679cb3279223f6d766fb9bbcce7e558afa836979e5e488f49c62/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/52e3c344eeacd929c3c2badfe16d2a02f7ee6fdf8a8268b876d7fe7041c4b696/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f8f9f8622f07fcff54264c02ccb918e1a1fc9b1d56b5b7bea5427aac2fca95ff/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1a98c176f83f46e35891f8acc0557b6e2c50f40a1c1b1fdd8d263d949b2270ce/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0efeb396c39c8c727530e0a9c01ce1650cc23aee2d009c4f4612daa11d5125c0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b16aaef2dec7ab4301233a0fd2186ceb2e0595b43c8ff285a129e861198f0b37/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ba97fd6bcdc0476cceb81ff7c0cfc2a7c2fa8629d508efa65610225956400851/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6d4ca1d28b081c33e35250dce179df9259f05ade5c450910eff814d71bd58c8f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8248e940483064dcb188dd9cc9d2fbffbba3da07e6e511b52cfc2305b43c1a77/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/6104027f232a4850a842843fa5bd1b6f5ed002aae3215dc58896d0b90acff03a/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/909551a695c4e91f2197c1eb736ea408c8f2771e7f4ec5283649d2cbce7546a5/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/13739b4041fda46ee9e22f9beff5d12169311353737c9c36767eeaee7391d978/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f1f2e2c6ebc272c7fad35817f642ad1e071e064fd8bebe1e1bfc244bc0477f08/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/e96c4e7125b7d420af4119ed6286bdb3c3709986412e61651447439113f29167/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/8065dfd2978ceb123814dcd0041f0fad5f5690ba45810ae79563fc932dfbfcd8/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7fb4163d9df107b802a904b7c4073eec4c76604bba615ee27a2a9cb73c875628/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/3edf330921922558311f259317a881f62004d640e5967bfbe0af7221478831cf/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/981ef9c3630ead35a93d4ec54a1f535247fe8d48db1a3e2e302e18cc80824802/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f5f4d60aa89d68306b9cb1f1c94dafaa30f8516810331181b7d6e085b97e56bf/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/01b16073c436504f0f97c087fdb4b0165d62112671ae3f1c8af7bcd146b33254/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/09abc62325209dd0a07c1fb6da1c2e72e8c0a52330f5e9e3fce1f462838a5b08/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/92ebe9ef0120c175a7446a777e333b0371b7e840e1865bffb59e89e0577b454e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e866d00b96ea32bf95e11dd69300d38458d6a3ac4f8ce1f396cbd779f8d976e4/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/9574eba3331a3b7bf7eb88dc4ba87acd711170893f9ad511c3df62b2248b60f5/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/90c1c3f74772de49dc4ed9ef5e8a614079b53afec71c88f16db3cb8d56af82f7/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/562cebfa8b23abb12fb1f47fc8d514cbc0c04cc2e918c25b24657a8679449773/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/8259a6125a125287f9de65223cb3dbaed06a41ffbbdafcf660de02e05222bccc/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/77728c29fa04090dbe113584c2d349dd6d41c74ac8d9b8da7c43e43d0a0e507d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/add1a93d827f2c5e89891ebf5392cd393aa8dc2e9f25cb8bfcc26256079b89b8/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d1448ddea473717dd11b006d521c8f233e90253d4a4d7b7fbe7fa9433ab8091a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d06b23c84fce0abb9e745fa6f99ddc31b278b44ff6a56847b482aa24c6027d81/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c5718a3c799fbcb3586e13be0dd4199c9bcbc25ac431e5609ab5ea4cbcfce74e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/7706a8c21e6f71ffd2882698c2362ccf030c5ebf27c24b3215171b037979fb0a/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/d2d883d4d5af51e2dc853050e84182d13d9a4e72ef94a4e6d9e6c53153bdffe2/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/fa06c3fa935d3ce9fa4aa57ed19f4add41e6b4ad7a38a578bc9f73d9da2d9da4/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/a71f0f1dd99e44d6b11c1d8c1d3432503df8a26d3b352e70d1f283423c446194/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/430178c43ba8a8a4aa127926494ff6411a47ed9060dd2d384ea71d3df2d107d1/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/3f7bc85780101f48d61b50870fd7f0c357c2854eca78b16cbd6856bfc1ca4425/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/4d0dd5abdf088c0f112da890ba9b0dc1a4d234d38b31f401a3ca504665867ffc/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/cefd4a75c863c305e10c9f773f76022a1b5bdadc8e1d752a00ff5b17805da131/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/266712d39d629ca24e9faf0a87fe0d9b2502a139f9c15ecd33ba3167bf616eb6/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/059a20cbd25aa2204c69bcb82db00f3fb9ffa9a46872e3db1582db391fbe5def/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/7fa0e410abb5bb06b5756449e06727556ba61e5d18adb65aedd17948f065d46c/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/44747a9da9af7050986affd008fc829377a4625b8c569a1e1d5e338ba610bba2/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/db9eea30e36f8d0c6c8ef80c420e6278b0040c5b573ae278a246a065b07b9f25/merged major:0 minor:55 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/618d7ac4bb2974843c3f757f1677873c02506b7ea6b609efa711b2ac5abc4217/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3b196e7694abac646cece1c559ad4974030d9be474b65c59d5c5cb795f35307e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/95430f826a26513661fe01950f20861cf6ebb1f444a8805ca8cb32edbcdc3682/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/756d5652495d1198630fd4651e6ac7db1c00eae6dea988f88d5701dfc48c9b89/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/bb4b9d19eb12db1ecfb6e16e5c9d136a75ec6c94187c14c85d772fc5b0bbf366/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/02b10506cf7eac32ab03385b9ab57b7dd6a9bc5637d705c860c5e78e4f2aa626/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/a3546062cdb9dfbb73ab58bddb918ab09bd73da6a2c0b169dedba7212e4e6115/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1f58fdcd4bb32ba420a4f8052ec53bed18ad87b1f3958c53f47266e5e55b3c74/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7465e7d4363791d593e4d68cc00441a5d49a84bb08eae01a5465654f0db0e0ff/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 23:53:52.134077 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.133514    5078 manager.go:217] Machine: {Timestamp:2026-10-30 23:53:52.133029073 +0000 UTC m=+0.079399782 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097678 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2458e279084deb2fe6e604e98b807e SystemUUID:ec2458e2-7908-4deb-2fe6-e604e98b807e BootID:6be6fa3e-79bf-4613-9e53-29d16925bb1f Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volumes/kubernetes.io~projected/kube-api-access-k2rft DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/543fceec0c8b2f801e752b9719182ce6c5930cb910b5d38a6bec764bebde24b3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4cac373-8436-4b03-a28b-0fa1bf8e7d13/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a286ab8f-3f4f-4621-8b10-3730b4e114f7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~projected/kube-api-access-npz6x DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c70641dbfafd9bfac31759ee3041f5af79c55071fff101a5d4daf58c06638b2a/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42ae59502822dbf55c1806cb0ef7cff61757499662a144052ce28b1aca20f1f3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58/volumes/kubernetes.io~projected/kube-api-access-859wz DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~projected/kube-api-access-8vrjb DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaa54632ff9fa07336eb53b95a1780aca536ad263959994a66ee93fe7dcd5143/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b25a3f44-1814-43a8-a01d-f98c870b7d9f/volumes/kubernetes.io~projected/kube-api-access-tbp2v DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be5dad9f-114f-4b12-bd51-217374c94fc0/volumes/kubernetes.io~projected/kube-api-access-nt4bc DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7eb92403-d07f-4bc4-bf34-4a15adff4863/volumes/kubernetes.io~projected/kube-api-access-fvx2l DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/658cf2080493fcde45b96bb4d3c5b6f6fc869bb3ae4130c2e0e65ca8da4e5cc4/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25170a0c-24d2-48b8-80fa-d2d2780a41a3/volumes/kubernetes.io~projected/kube-api-access-xkwq4 DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fe484ab6-ac95-4849-acc0-37636302d300/volumes/kubernetes.io~projected/kube-api-access-r4557 DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa85bf8b-61bf-4bf1-b820-05215a7eadf3/volumes/kubernetes.io~projected/kube-api-access-cb8kf DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1090e22-f04c-4bda-b340-0966209254af/volumes/kubernetes.io~projected/kube-api-access-h2dmv DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/964f2946-7176-4f03-b41f-03bd65aa8346/volumes/kubernetes.io~projected/kube-api-access-pqjsr DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/805ad22b7c4299f7115fc2597436c5b6c1ea34ebd01350d792fe4399e7e14ba4/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b213ff9-3e00-426a-901f-e5ca105d37e8/volumes/kubernetes.io~projected/kube-api-access-pqdzc DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1384f8a-a2ad-4e55-b6e5-14ab134fa489/volumes/kubernetes.io~projected/kube-api-access-95p8n DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4498189b-b2cb-46e0-b20f-926be273f49a/volumes/kubernetes.io~projected/kube-api-access-zb26f DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51bd9499-546b-4b4e-8350-7501c25b0ab2/volumes/kubernetes.io~projected/kube-api-access-qwggr DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a286ab8f-3f4f-4621-8b10-3730b4e114f7/volumes/kubernetes.io~projected/kube-api-access-dhmm2 DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e73a9af6-0664-409b-a096-83a990ec98ce/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e712089b3aca389d206f6b5bceadd50b13d3cc34605e16bbe9222d860b9c731/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c63bdce51935f10978dca592caebae675494c31b509d80e1d7c4a933a8b350d9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/362b6ff8-9785-49c5-8704-db4c94a603d9/volumes/kubernetes.io~projected/kube-api-access-mhm77 DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/560aedf5587cb1bbec4f517bfe9d369a10c3b48731c4bde64aa345c07e596372/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0589c1c8271028a4082fe94ca3a9081fe9d7354d7ef976bcd06c325bedd2daf2/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5cdbbc3a4a37e4239a50b2004665d95f0e6099c8b6fdf68ff822913dd5bc4e2/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/daf52a97-c6ca-4fba-a77b-67841614c162/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e19fb8ace6bbe6512ce9a1a0a313f1271f12b3a3686238d2f88bee2fa087955/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fd4aa0b5415a56dc4be0df0faceb3013764769671799e995241efc575805a6a/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7057aab3-480b-4bd1-81cc-e100e33b35d5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72c64997-e326-411c-9f16-8ad1671ef849/volumes/kubernetes.io~projected/kube-api-access-q86sq DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bce9a80eb98cbd6fbf93d449717b0e220a1e47b6e8a20f97d6a01cb2dbf5556/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/volumes/kubernetes.io~projected/kube-api-access-v8tst DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dab3bc3e9f62882b52123bd6a9c37f123fc1759017ca8eb67ff2078c7a9d3527/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72c64997-e326-411c-9f16-8ad1671ef849/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~projected/kube-api-access-ng52f DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4cac373-8436-4b03-a28b-0fa1bf8e7d13/volumes/kubernetes.io~projected/kube-api-access-fl8h7 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4739b9ea84bd9a227b5d6b9b86cc4e049bfaf4d29620519c80eb32cbcc0272e/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3bbf2354-7b35-4374-adc3-e694b71f6e9e/volumes/kubernetes.io~projected/kube-api-access-sgh87 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/567d58b9-e17a-41fd-a5e2-6ec9ff24597d/volumes/kubernetes.io~projected/kube-api-access-wkdnh DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eb92403-d07f-4bc4-bf34-4a15adff4863/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/500490ba-4b0f-4e40-9f8e-3906d62c458c/volumes/kubernetes.io~projected/kube-api-access-f5zfx DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfa573d4014e5e756afb94299f6ac257e16ce313f246be88d38338066053206c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6e24b7db-f2a1-4500-9d82-963292cb7765/volumes/kubernetes.io~projected/kube-api-access-4jpsf DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7/volumes/kubernetes.io~projected/kube-api-access-fgpwz DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf822d012c8578808fc8a406edc23586a4c395b0ac628606d0eef4b2d5ba348a/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8334583188f22aff5f1f760518baeed1578ef43e85516cbdf074c091bd3d8fcf/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88aedb21-8c5c-457b-a5c1-b778c33565c3/volumes/kubernetes.io~projected/kube-api-access-ltt9r DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6789dc-0c9b-4507-9615-93c3ec632af2/volumes/kubernetes.io~projected/kube-api-access-t4qs8 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/volumes/kubernetes.io~projected/kube-api-access-g8586 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21c3876082cc31e3a38531453206ed9737208e3e2106d17e2849a58a92fc8b1e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5b193c2189e0a2c236bea7075be989f101ac299bb71b4b39fdea800adf16a11/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1a8190-b44f-41ac-8f0f-5020e6e27c18/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/daf52a97-c6ca-4fba-a77b-67841614c162/volumes/kubernetes.io~projected/kube-api-access-x6w5v DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/197fd1d3503c9951740ae5689f7f05612e200977a5ad360a5dadc36d107c681d/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03d20f01c85f8e4e0e5b6ccf18b687241934c3d6df07b64a8948c7b928ab3e45/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0589c1c8271028a MacAddress:3a:41:7b:07:ff:23 Speed:10000 Mtu:8901} {Name:197fd1d3503c995 MacAddress:12:3c:de:bb:18:95 Speed:10000 Mtu:8901} {Name:21c3876082cc31e MacAddress:b2:50:fe:e9:d6:9b Speed:10000 Mtu:8901} {Name:2e712089b3aca38 MacAddress:06:4a:a9:1f:9a:32 Speed:10000 Mtu:8901} {Name:3fd4aa0b5415a56 MacAddress:ea:84:f0:ad:7b:00 Speed:10000 Mtu:8901} {Name:42ae59502822dbf MacAddress:92:fa:73:19:6e:8c Speed:10000 Mtu:8901} {Name:560aedf5587cb1b MacAddress:3a:41:f5:2f:89:58 Speed:10000 Mtu:8901} {Name:5b5c76ff8667d1a MacAddress:66:88:aa:10:03:e9 Speed:10000 Mtu:8901} {Name:658cf2080493fcd MacAddress:f6:da:d9:72:9a:40 Speed:10000 Mtu:8901} {Name:805ad22b7c4299f MacAddress:72:92:ce:01:10:da Speed:10000 Mtu:8901} {Name:8334583188f22af MacAddress:d2:01:d6:42:bf:eb Speed:10000 Mtu:8901} {Name:8e19fb8ace6bbe6 MacAddress:02:f9:d4:15:66:0e Speed:10000 Mtu:8901} {Name:a4739b9ea84bd9a MacAddress:f2:a7:47:b9:cc:08 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:38:1f:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:2c:09:fb:c8:ba Speed:0 Mtu:8901} {Name:c5cdbbc3a4a37e4 MacAddress:ee:0c:74:88:b2:14 Speed:10000 Mtu:8901} {Name:c70641dbfafd9bf MacAddress:e6:d1:97:97:5a:e1 Speed:10000 Mtu:8901} {Name:cf822d012c85788 MacAddress:22:a5:0e:1c:4e:f9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:38:1f:55 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ae:69:a9:d5:2f:17 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:40:e7:65:e4:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:53:52.134077 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.133853    5078 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:53:52.134077 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.133895    5078 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:53:52.134310 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134177    5078 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:53:52.134310 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134194    5078 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-16.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:53:52.134375 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134314    5078 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:53:52.134375 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134333    5078 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:53:52.134375 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134343    5078 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:53:52.134375 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134355    5078 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:53:52.134375 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134374    5078 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:53:52.134591 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134552    5078 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:53:52.134681 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134671    5078 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:53:52.134719 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134710    5078 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:53:52.134750 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134729    5078 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:53:52.134750 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134740    5078 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:53:52.134750 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134748    5078 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:53:52.134847 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.134757    5078 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:53:52.136984 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.136955    5078 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:53:52.136984 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.136977    5078 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:53:52.137699 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.137679    5078 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:53:52.140863 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.140848    5078 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:53:52.140863 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.140863    5078 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:53:52.140967 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.140958    5078 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 23:53:52.141146 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141135    5078 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:53:52.141186 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141165    5078 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:53:52.141265 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141256    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:53:52.141299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141270    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:53:52.141299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141276    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:53:52.141299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141282    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:53:52.141299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141288    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:53:52.141299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141293    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:53:52.141299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141299    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141309    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141317    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141322    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141330    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141338    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141363    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:53:52.141685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.141373    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:53:52.145105 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.145089    5078 server.go:1263] "Started kubelet"
Oct 30 23:53:52.145300 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.145230    5078 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:53:52.145368 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.145343    5078 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:53:52.145420 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.145361    5078 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:53:52.145679 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.145660    5078 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:53:52.145805 ip-10-0-143-16 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:53:52.146705 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.146681    5078 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:53:52.149665 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.149642    5078 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:53:52.151222 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.151198    5078 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:53:52.151222 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.151221    5078 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:53:52.151378 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.151273    5078 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:53:52.151580 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.151542    5078 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-east-2.compute.internal\" not found"
Oct 30 23:53:52.151657 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.151611    5078 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:53:52.151657 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.151637    5078 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:10 +0000 UTC" deadline="2028-05-20 03:38:10.257852521 +0000 UTC"
Oct 30 23:53:52.151657 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.151655    5078 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13611h44m18.106200557s"
Oct 30 23:53:52.152672 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152653    5078 factory.go:153] Registering CRI-O factory
Oct 30 23:53:52.152672 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152674    5078 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:53:52.152797 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152734    5078 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:53:52.152797 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152743    5078 factory.go:55] Registering systemd factory
Oct 30 23:53:52.152797 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152751    5078 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:53:52.152797 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152767    5078 factory.go:103] Registering Raw factory
Oct 30 23:53:52.152797 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.152782    5078 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:53:52.153136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.153123    5078 manager.go:319] Starting recovery of all containers
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160521    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500490ba-4b0f-4e40-9f8e-3906d62c458c" volumeName="kubernetes.io/projected/500490ba-4b0f-4e40-9f8e-3906d62c458c-kube-api-access-f5zfx" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160597    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" volumeName="kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-trusted-ca" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160661    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65fab40a-d3b0-4bf1-9ab3-396dfccebe5f" volumeName="kubernetes.io/secret/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-agent-certs" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160708    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e24b7db-f2a1-4500-9d82-963292cb7765" volumeName="kubernetes.io/projected/6e24b7db-f2a1-4500-9d82-963292cb7765-kube-api-access-4jpsf" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160760    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160815    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1090e22-f04c-4bda-b340-0966209254af" volumeName="kubernetes.io/projected/d1090e22-f04c-4bda-b340-0966209254af-kube-api-access-h2dmv" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160834    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6789dc-0c9b-4507-9615-93c3ec632af2" volumeName="kubernetes.io/projected/5f6789dc-0c9b-4507-9615-93c3ec632af2-kube-api-access-t4qs8" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160845    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1090e22-f04c-4bda-b340-0966209254af" volumeName="kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160907    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" volumeName="kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-default-certificate" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.160972    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf52a97-c6ca-4fba-a77b-67841614c162" volumeName="kubernetes.io/secret/daf52a97-c6ca-4fba-a77b-67841614c162-serving-cert" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161006    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb92403-d07f-4bc4-bf34-4a15adff4863" volumeName="kubernetes.io/projected/7eb92403-d07f-4bc4-bf34-4a15adff4863-kube-api-access-fvx2l" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161020    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a286ab8f-3f4f-4621-8b10-3730b4e114f7" volumeName="kubernetes.io/configmap/a286ab8f-3f4f-4621-8b10-3730b4e114f7-config" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161033    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" volumeName="kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161063    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e73a9af6-0664-409b-a096-83a990ec98ce" volumeName="kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161082    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" volumeName="kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-config" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161115    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="567d58b9-e17a-41fd-a5e2-6ec9ff24597d" volumeName="kubernetes.io/projected/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-api-access-wkdnh" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161123    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" volumeName="kubernetes.io/secret/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-serving-cert" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161145    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c64997-e326-411c-9f16-8ad1671ef849" volumeName="kubernetes.io/configmap/72c64997-e326-411c-9f16-8ad1671ef849-signing-cabundle" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161160    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" volumeName="kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-config" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161172    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a91f2be6-7bda-48a0-a0fc-8d5da8469dd7" volumeName="kubernetes.io/projected/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-kube-api-access-fgpwz" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161198    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa85bf8b-61bf-4bf1-b820-05215a7eadf3" volumeName="kubernetes.io/projected/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-kube-api-access-cb8kf" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161234    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161251    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161258    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161277    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161313    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" volumeName="kubernetes.io/projected/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-kube-api-access-v8tst" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161320    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e24b7db-f2a1-4500-9d82-963292cb7765" volumeName="kubernetes.io/configmap/6e24b7db-f2a1-4500-9d82-963292cb7765-serviceca" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161365    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c64997-e326-411c-9f16-8ad1671ef849" volumeName="kubernetes.io/projected/72c64997-e326-411c-9f16-8ad1671ef849-kube-api-access-q86sq" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161388    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="964f2946-7176-4f03-b41f-03bd65aa8346" volumeName="kubernetes.io/configmap/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-config" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161412    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a286ab8f-3f4f-4621-8b10-3730b4e114f7" volumeName="kubernetes.io/projected/a286ab8f-3f4f-4621-8b10-3730b4e114f7-kube-api-access-dhmm2" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161433    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb92403-d07f-4bc4-bf34-4a15adff4863" volumeName="kubernetes.io/secret/7eb92403-d07f-4bc4-bf34-4a15adff4863-cert" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161453    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a286ab8f-3f4f-4621-8b10-3730b4e114f7" volumeName="kubernetes.io/secret/a286ab8f-3f4f-4621-8b10-3730b4e114f7-serving-cert" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161478    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161547    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a91f2be6-7bda-48a0-a0fc-8d5da8469dd7" volumeName="kubernetes.io/configmap/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-telemetry-config" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.161563    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" volumeName="kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-snapshots" seLinuxMountContext=""
Oct 30 23:53:52.163091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.162953    5078 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.164956    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" volumeName="kubernetes.io/projected/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-kube-api-access-fl8h7" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.164992    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" volumeName="kubernetes.io/projected/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-kube-api-access-g8586" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165005    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" volumeName="kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165037    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b213ff9-3e00-426a-901f-e5ca105d37e8" volumeName="kubernetes.io/projected/6b213ff9-3e00-426a-901f-e5ca105d37e8-kube-api-access-pqdzc" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165050    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf52a97-c6ca-4fba-a77b-67841614c162" volumeName="kubernetes.io/configmap/daf52a97-c6ca-4fba-a77b-67841614c162-config" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165062    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe484ab6-ac95-4849-acc0-37636302d300" volumeName="kubernetes.io/projected/fe484ab6-ac95-4849-acc0-37636302d300-kube-api-access-r4557" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165074    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165086    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b949609-5560-4b48-a5b7-229d7430d7c0" volumeName="kubernetes.io/configmap/1b949609-5560-4b48-a5b7-229d7430d7c0-nginx-conf" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165097    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="362b6ff8-9785-49c5-8704-db4c94a603d9" volumeName="kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165146    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bbf2354-7b35-4374-adc3-e694b71f6e9e" volumeName="kubernetes.io/projected/3bbf2354-7b35-4374-adc3-e694b71f6e9e-kube-api-access-sgh87" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165159    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65fab40a-d3b0-4bf1-9ab3-396dfccebe5f" volumeName="kubernetes.io/configmap/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-konnectivity-ca" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165181    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be5dad9f-114f-4b12-bd51-217374c94fc0" volumeName="kubernetes.io/projected/be5dad9f-114f-4b12-bd51-217374c94fc0-kube-api-access-nt4bc" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165193    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" volumeName="kubernetes.io/configmap/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-config-volume" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165204    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" volumeName="kubernetes.io/empty-dir/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-tmp-dir" seLinuxMountContext=""
Oct 30 23:53:52.165250 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165217    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" volumeName="kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-env-overrides" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165260    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165274    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7057aab3-480b-4bd1-81cc-e100e33b35d5" volumeName="kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-tmp" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165289    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="964f2946-7176-4f03-b41f-03bd65aa8346" volumeName="kubernetes.io/projected/964f2946-7176-4f03-b41f-03bd65aa8346-kube-api-access-pqjsr" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165303    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf52a97-c6ca-4fba-a77b-67841614c162" volumeName="kubernetes.io/projected/daf52a97-c6ca-4fba-a77b-67841614c162-kube-api-access-x6w5v" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165314    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165329    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" volumeName="kubernetes.io/projected/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-kube-api-access-k2rft" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165379    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7057aab3-480b-4bd1-81cc-e100e33b35d5" volumeName="kubernetes.io/projected/7057aab3-480b-4bd1-81cc-e100e33b35d5-kube-api-access-8vrjb" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165391    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88aedb21-8c5c-457b-a5c1-b778c33565c3" volumeName="kubernetes.io/projected/88aedb21-8c5c-457b-a5c1-b778c33565c3-kube-api-access-ltt9r" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165417    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" volumeName="kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-service-ca-bundle" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165445    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" volumeName="kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-stats-auth" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165503    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4498189b-b2cb-46e0-b20f-926be273f49a" volumeName="kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165515    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4498189b-b2cb-46e0-b20f-926be273f49a" volumeName="kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165527    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" volumeName="kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165537    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" volumeName="kubernetes.io/secret/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-serving-cert" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165551    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" volumeName="kubernetes.io/projected/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-kube-api-access-95p8n" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165562    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa85bf8b-61bf-4bf1-b820-05215a7eadf3" volumeName="kubernetes.io/configmap/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-iptables-alerter-script" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165589    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" volumeName="kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165601    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4498189b-b2cb-46e0-b20f-926be273f49a" volumeName="kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165614    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7057aab3-480b-4bd1-81cc-e100e33b35d5" volumeName="kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-tuned" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165643    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165655    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165667    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" volumeName="kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-tmp" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165679    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25170a0c-24d2-48b8-80fa-d2d2780a41a3" volumeName="kubernetes.io/projected/25170a0c-24d2-48b8-80fa-d2d2780a41a3-kube-api-access-xkwq4" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165691    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165716    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="567d58b9-e17a-41fd-a5e2-6ec9ff24597d" volumeName="kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165729    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25a3f44-1814-43a8-a01d-f98c870b7d9f" volumeName="kubernetes.io/projected/b25a3f44-1814-43a8-a01d-f98c870b7d9f-kube-api-access-tbp2v" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165739    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" volumeName="kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165740    5078 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165753    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f9a68d-9d3a-4032-9f05-821e5563894a" volumeName="kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165766    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" volumeName="kubernetes.io/secret/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165778    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab1743-a264-4439-a1a1-fd147ebd1248" volumeName="kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165791    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51bd9499-546b-4b4e-8350-7501c25b0ab2" volumeName="kubernetes.io/projected/51bd9499-546b-4b4e-8350-7501c25b0ab2-kube-api-access-qwggr" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165814    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="567d58b9-e17a-41fd-a5e2-6ec9ff24597d" volumeName="kubernetes.io/empty-dir/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-data-volume" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165825    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c64997-e326-411c-9f16-8ad1671ef849" volumeName="kubernetes.io/secret/72c64997-e326-411c-9f16-8ad1671ef849-signing-key" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165838    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1090e22-f04c-4bda-b340-0966209254af" volumeName="kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-multus-daemon-config" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165852    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" volumeName="kubernetes.io/projected/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-kube-api-access-859wz" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165865    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bbf2354-7b35-4374-adc3-e694b71f6e9e" volumeName="kubernetes.io/empty-dir/3bbf2354-7b35-4374-adc3-e694b71f6e9e-tmp-dir" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165884    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4498189b-b2cb-46e0-b20f-926be273f49a" volumeName="kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165896    5078 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:53:52.165990 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.165904    5078 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:53:52.168780 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.168755    5078 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:52.170717 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.170621    5078 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 23:53:52.173678 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.173657    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:53:52.174730 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.174707    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:53:52.184622 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.184573    5078 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:53:52.186343 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.186323    5078 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:53:52.186343 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.186345    5078 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:53:52.186522 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.186369    5078 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:53:52.186522 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.186414    5078 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:53:52.194847 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.193249    5078 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:53:52.221917 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.221896    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:53:52.222031 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.221925    5078 generic.go:356] "Generic (PLEG): container finished" podID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160" exitCode=255
Oct 30 23:53:52.230231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.230209    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="b80be36abb6e7ca0b6b972aa77c6b7b69c34e354b38cd327ba686aa605530aff" exitCode=0
Oct 30 23:53:52.230231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.230229    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="58743ea02cf0a206407ea8dd831f449b2aba136716415162f28b9a19612626d4" exitCode=0
Oct 30 23:53:52.230396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.230235    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="3354262f7bcf0e8ac775cd5ac943ea41c4fad62443f091995bd521b9aaaeb5a7" exitCode=0
Oct 30 23:53:52.230396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.230241    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="e614d2eafab726af5128ab3b28a5a1e4177e1283b7ac3d16950c92324f340048" exitCode=0
Oct 30 23:53:52.230396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.230248    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="242dcf40320fd6ec74a851427390243a49a79fc4c004d8eec21da970a39c3276" exitCode=0
Oct 30 23:53:52.230396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.230253    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58" containerID="416dbee70618b84d1eb8e9b0696a20e6b1c0587c5582071e12199a2bc9796586" exitCode=0
Oct 30 23:53:52.233665 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.233637    5078 generic.go:356] "Generic (PLEG): container finished" podID="f6d13e136b05ce96356f6b9a00f72419" containerID="d2633c913d8d14d20dc296c29382ae191fc6dfde0ae2c41de18a5b6671877c65" exitCode=0
Oct 30 23:53:52.257621 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.257577    5078 manager.go:324] Recovery completed
Oct 30 23:53:52.279692 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.279674    5078 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:53:52.279692 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.279689    5078 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:53:52.279816 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.279705    5078 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:53:52.279847 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.279813    5078 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 23:53:52.280281 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.279823    5078 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 23:53:52.280343 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280334    5078 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 23:53:52.280373 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280343    5078 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 23:53:52.280373 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280349    5078 policy_none.go:50] "Start"
Oct 30 23:53:52.280373 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280358    5078 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:53:52.280373 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280370    5078 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:53:52.280533 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280524    5078 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 23:53:52.280563 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280534    5078 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 23:53:52.280563 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.280540    5078 policy_none.go:44] "Start"
Oct 30 23:53:52.284958 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.284929    5078 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:53:52.286910 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.286885    5078 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 23:53:52.287569 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.287553    5078 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:53:52.287654 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.287576    5078 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:53:52.287654 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.287585    5078 server.go:87] "Starting device plugin registration server"
Oct 30 23:53:52.287881 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.287868    5078 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:53:52.287946 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.287881    5078 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:53:52.288004 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.287972    5078 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:53:52.288113 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.288071    5078 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:53:52.288178 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.288120    5078 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:53:52.288178 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.288127    5078 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:53:52.288275 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.288176    5078 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:53:52.288109031Z","UUID":"3b11374d-3991-418d-95b2-5f2250e9a5c2","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:53:52.289380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.289362    5078 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:53:52.289490 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.289386    5078 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:53:52.289772 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.289750    5078 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:53:52.388398 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.388375    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:53:52.389518 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.389503    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:53:52.389573 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.389531    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:53:52.389573 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.389542    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:53:52.389627 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.389583    5078 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.406299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.406277    5078 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.406367 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.406334    5078 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.487182 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487137    5078 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"]
Oct 30 23:53:52.487828 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487792    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"f6d13e136b05ce96356f6b9a00f72419","Type":"ContainerStarted","Data":"cdf6b0bcba652d08360b814c0c6c9c0c65f820d0fc5cfba5b01c982e8e61e0a1"}
Oct 30 23:53:52.487934 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487838    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"f6d13e136b05ce96356f6b9a00f72419","Type":"ContainerDied","Data":"d2633c913d8d14d20dc296c29382ae191fc6dfde0ae2c41de18a5b6671877c65"}
Oct 30 23:53:52.487934 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487859    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"f6d13e136b05ce96356f6b9a00f72419","Type":"ContainerStarted","Data":"b8d8e292407aa5d3e106076bbcb01df35a14eb1fc2b1c86c63575411f426eb09"}
Oct 30 23:53:52.487934 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487897    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"fe6922bb420b9fc1aa0231980b4f658c","Type":"ContainerStarted","Data":"88a5b208d29e648818ff72025ce3d666e8ca81f32e51ca23218e8ff2824bfaed"}
Oct 30 23:53:52.487934 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487915    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal" event={"ID":"fe6922bb420b9fc1aa0231980b4f658c","Type":"ContainerStarted","Data":"f94efca16a5ae691c94291d83f4b2cd46e2fa5aeeca28f4be33fc0521f3b7baf"}
Oct 30 23:53:52.487934 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487808    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.488143 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.487965    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.503975 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.503957    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:53:52.504048 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.504016    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.504511 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.504495    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:53:52.504548 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:52.504540    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.572585 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.572531    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.572585 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.572559    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.572585 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.572574    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe6922bb420b9fc1aa0231980b4f658c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"fe6922bb420b9fc1aa0231980b4f658c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.673126 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.673101    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.673206 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.673130    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.673206 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.673155    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe6922bb420b9fc1aa0231980b4f658c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"fe6922bb420b9fc1aa0231980b4f658c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.673206 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.673192    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe6922bb420b9fc1aa0231980b4f658c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"fe6922bb420b9fc1aa0231980b4f658c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.673310 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.673228    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:52.673310 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:52.673231    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6d13e136b05ce96356f6b9a00f72419-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" (UID: \"f6d13e136b05ce96356f6b9a00f72419\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:53.135214 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.135184    5078 apiserver.go:52] "Watching apiserver"
Oct 30 23:53:53.151002 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.150970    5078 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:53:53.151908 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.151877    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp","openshift-insights/insights-operator-7c4986d449-mb6hw","openshift-service-ca/service-ca-5fffc97bdf-84r2z","openshift-image-registry/image-registry-6449bcf677-rqxsk","openshift-image-registry/image-registry-674dd678bd-xdv8h","openshift-ingress-canary/ingress-canary-mrwcz","openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6","openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms","openshift-insights/insights-runtime-extractor-hgtpq","kube-system/kas-connection-checker-685895c7c9-69tlv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv","kube-system/kas-connection-checker-685895c7c9-2qk9k","openshift-cluster-node-tuning-operator/tuned-db4cd","openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr","openshift-image-registry/node-ca-bndnt","kube-system/konnectivity-agent-s8g6n","openshift-ingress/router-default-74cc75bf99-fkb87","openshift-network-console/networking-console-plugin-86c478bb88-k7rhq","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal","openshift-multus/multus-4f6sj","openshift-multus/multus-additional-cni-plugins-7bbnp","openshift-console-operator/console-operator-7cd56b8dbc-zzxg7","openshift-multus/network-metrics-daemon-vl6sr","openshift-ovn-kubernetes/ovnkube-node-hqmv6","kube-system/global-pull-secret-syncer-ktssl","kube-system/kas-connection-checker-685895c7c9-kgs9w","openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8","openshift-dns/node-resolver-ccfzb","openshift-dns/dns-default-shskm","openshift-insights/periodic-gathering-rnrwh-lb8r2","openshift-network-diagnostics/network-check-target-2qs9j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79","openshift-network-operator/iptables-alerter-jmmdd"]
Oct 30 23:53:53.156815 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.156773    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.157685 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.157664    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.158136 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.158118    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.158872 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.158857    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.159239 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.159207    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.159841 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.159824    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.160554 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160522    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.160554 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160537    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.160683 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160560    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.160683 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160578    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 23:53:53.160683 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160560    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 23:53:53.160683 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160639    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.160870 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.160719    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.163456 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.163436    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.164623 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.164602    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 23:53:53.165133 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.165117    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.165585 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.165568    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.165585 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.165572    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.165715 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.165677    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.165823 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.165809    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 23:53:53.165823 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.165809    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 23:53:53.168923 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.168890    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.169624 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.169605    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2gpl5\""
Oct 30 23:53:53.169835 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.169820    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 23:53:53.170913 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.170895    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 23:53:53.171168 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.171147    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z8jlp\""
Oct 30 23:53:53.171263 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.171246    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 23:53:53.171349 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.171334    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.171919 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.171899    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q4t65\""
Oct 30 23:53:53.177774 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.177752    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 23:53:53.177774 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.177752    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 23:53:53.177940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.177924    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 23:53:53.179275 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179252    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 23:53:53.179275 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179261    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.179474 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179286    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.179474 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179260    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.179474 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179306    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:53:53.179474 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179308    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:53:53.179474 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179335    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.179630 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179486    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.179888 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179862    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:53:53.179982 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179952    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:53:53.179982 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179961    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.179986    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.180010    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.180017    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.180045    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.180027    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.180036    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:53:53.180116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.180083    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:53:53.181754 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181712    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.181754 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181720    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181774    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181795    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181794    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181826    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181832    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181796    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.181907 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181871    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181914    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181918    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181920    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.181932    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r8slq\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182107    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182225    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182268    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182326    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 23:53:53.182340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182352    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182423    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182453    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182489    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182501    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182504    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182549    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182558    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182579    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182606    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7dvpv\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182609    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182634    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182620    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8s8qp\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182646    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q6gjg\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182671    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-427wx\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182683    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182696    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182719    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182725    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.182865 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.182744    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.183923 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.183910    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.183963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.183931    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.184244 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184230    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2wwf\""
Oct 30 23:53:53.184336 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184320    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wjrgq\""
Oct 30 23:53:53.184406 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184344    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4mxgc\""
Oct 30 23:53:53.184406 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184365    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:53:53.184406 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184367    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9xt4\""
Oct 30 23:53:53.184406 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184373    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\""
Oct 30 23:53:53.184564 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184409    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhccc\""
Oct 30 23:53:53.184564 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.184438    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:53:53.185034 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185015    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\""
Oct 30 23:53:53.185034 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185029    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gq6b\""
Oct 30 23:53:53.185193 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185017    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 23:53:53.185193 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185061    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tb4c7\""
Oct 30 23:53:53.185193 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185129    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vb9sd\""
Oct 30 23:53:53.185193 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185146    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z44sw\""
Oct 30 23:53:53.185193 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185174    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bdhrq\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185280    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4xs9v\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185289    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185309    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sdzl9\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185319    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185319    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185320    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185313    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185351    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:53:53.185396 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.185367    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g4klp\""
Oct 30 23:53:53.186662 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.186646    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:53:53.186768 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.186753    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pr9l6\""
Oct 30 23:53:53.187921 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.187908    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:53:53.188002 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.187948    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.189278 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.189265    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 23:53:53.193825 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.193808    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:53:53.194281 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.194267    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 23:53:53.249418 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.249401    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:53.249549 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.249537    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:53.254280 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.254265    5078 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:53:53.263028 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.263015    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:53:53.263089 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.263058    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:53.263227 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.263213    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:53:53.263267 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.263254    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal"
Oct 30 23:53:53.276543 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276521    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysconfig\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.276642 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276558    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-ovn\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.276642 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276589    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e24b7db-f2a1-4500-9d82-963292cb7765-serviceca\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.276749 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276730    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4557\" (UniqueName: \"kubernetes.io/projected/fe484ab6-ac95-4849-acc0-37636302d300-kube-api-access-r4557\") pod \"network-check-source-6f78d59f99-rw5ms\" (UID: \"fe484ab6-ac95-4849-acc0-37636302d300\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:53.276788 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276773    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-sys-fs\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.276839 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276796    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-sys\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.276901 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276877    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-etc-kubernetes\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.276958 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276922    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-host-slash\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.276958 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276882    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e24b7db-f2a1-4500-9d82-963292cb7765-serviceca\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.277032 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276954    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a286ab8f-3f4f-4621-8b10-3730b4e114f7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:53.277032 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.276990    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-default-certificate\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.277130 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277078    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.277179 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277159    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.277231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277161    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a286ab8f-3f4f-4621-8b10-3730b4e114f7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:53.277231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277163    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-default-certificate\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.277231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277208    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-var-lib-kubelet\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277233    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-multus-daemon-config\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277250    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbp2v\" (UniqueName: \"kubernetes.io/projected/b25a3f44-1814-43a8-a01d-f98c870b7d9f-kube-api-access-tbp2v\") pod \"volume-data-source-validator-58c9f5f46d-sf4kv\" (UID: \"b25a3f44-1814-43a8-a01d-f98c870b7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277264    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-systemd-units\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277284    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-netns\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277307    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rft\" (UniqueName: \"kubernetes.io/projected/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-kube-api-access-k2rft\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277340    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-serving-cert\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.277385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277376    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277389    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277485    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95p8n\" (UniqueName: \"kubernetes.io/projected/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-kube-api-access-95p8n\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277571    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-serving-cert\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277620    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-multus-daemon-config\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277645    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltt9r\" (UniqueName: \"kubernetes.io/projected/88aedb21-8c5c-457b-a5c1-b778c33565c3-kube-api-access-ltt9r\") pod \"migrator-66df5d496b-fjlhr\" (UID: \"88aedb21-8c5c-457b-a5c1-b778c33565c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277673    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4bc\" (UniqueName: \"kubernetes.io/projected/be5dad9f-114f-4b12-bd51-217374c94fc0-kube-api-access-nt4bc\") pod \"kas-connection-checker-685895c7c9-2qk9k\" (UID: \"be5dad9f-114f-4b12-bd51-217374c94fc0\") " pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277691    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpsf\" (UniqueName: \"kubernetes.io/projected/6e24b7db-f2a1-4500-9d82-963292cb7765-kube-api-access-4jpsf\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.277735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277711    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6w5v\" (UniqueName: \"kubernetes.io/projected/daf52a97-c6ca-4fba-a77b-67841614c162-kube-api-access-x6w5v\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277791    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277832    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-conf\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277874    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-node-log\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277903    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-serving-cert\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.277958    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278007    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-tmp-dir\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278034    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278064    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86sq\" (UniqueName: \"kubernetes.io/projected/72c64997-e326-411c-9f16-8ad1671ef849-kube-api-access-q86sq\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278073    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-serving-cert\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278086    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-agent-certs\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278097    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-tmp-dir\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.278101 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278111    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278154    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278184    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278211    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278252    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278281    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-config-volume\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278331    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf52a97-c6ca-4fba-a77b-67841614c162-serving-cert\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278397    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278440    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-agent-certs\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278507    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf52a97-c6ca-4fba-a77b-67841614c162-serving-cert\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278507    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-config-volume\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278530    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278558    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278565    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278588    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.278635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278622    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278748    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-kubelet-config\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278765    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-dbus\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278769    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278811    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278845    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-data-volume\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278884    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-host\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278911    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-data-volume\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278915    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-netd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278951    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.278952    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279044    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-tmp\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279081    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b949609-5560-4b48-a5b7-229d7430d7c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279097    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-kubelet\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279113    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-env-overrides\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279137    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-tmp\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.279164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279152    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-tmp\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279197    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bbf2354-7b35-4374-adc3-e694b71f6e9e-tmp-dir\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279233    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279249    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279280    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-tmp\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279289    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b949609-5560-4b48-a5b7-229d7430d7c0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279304    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-env-overrides\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-bin\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279375    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bbf2354-7b35-4374-adc3-e694b71f6e9e-tmp-dir\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279410    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279481    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-script-lib\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279510    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-socket-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279528    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279543    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovn-node-metrics-cert\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279565    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279666    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279679    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279695    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-log-socket\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279707    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-script-lib\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279712    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovn-node-metrics-cert\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279720    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279762    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-tuned\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279793    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279839    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwggr\" (UniqueName: \"kubernetes.io/projected/51bd9499-546b-4b4e-8350-7501c25b0ab2-kube-api-access-qwggr\") pod \"kas-connection-checker-685895c7c9-69tlv\" (UID: \"51bd9499-546b-4b4e-8350-7501c25b0ab2\") " pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279843    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.279864 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279864    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-tuned\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.279903    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280005    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280091    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280194    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf52a97-c6ca-4fba-a77b-67841614c162-config\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280213    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280238    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280232    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-multus\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280265    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e73a9af6-0664-409b-a096-83a990ec98ce-original-pull-secret\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280283    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-kube-api-access-859wz\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-device-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280332    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-config\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280338    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280365    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a286ab8f-3f4f-4621-8b10-3730b4e114f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280393    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwq4\" (UniqueName: \"kubernetes.io/projected/25170a0c-24d2-48b8-80fa-d2d2780a41a3-kube-api-access-xkwq4\") pod \"kas-connection-checker-685895c7c9-kgs9w\" (UID: \"25170a0c-24d2-48b8-80fa-d2d2780a41a3\") " pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280419    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf52a97-c6ca-4fba-a77b-67841614c162-config\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280418    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-k8s-cni-cncf-io\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280497    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280527    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-bin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280579    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-ovnkube-config\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280633    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-iptables-alerter-script\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280658    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-etc-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280690    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280699    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a286ab8f-3f4f-4621-8b10-3730b4e114f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280742    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280765    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280781    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-multus-certs\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280825    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmm2\" (UniqueName: \"kubernetes.io/projected/a286ab8f-3f4f-4621-8b10-3730b4e114f7-kube-api-access-dhmm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280865    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-iptables-alerter-script\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280893    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bbf2354-7b35-4374-adc3-e694b71f6e9e-hosts-file\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280894    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280930    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgh87\" (UniqueName: \"kubernetes.io/projected/3bbf2354-7b35-4374-adc3-e694b71f6e9e-kube-api-access-sgh87\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.280940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.280956    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjsr\" (UniqueName: \"kubernetes.io/projected/964f2946-7176-4f03-b41f-03bd65aa8346-kube-api-access-pqjsr\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281073    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-systemd\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281108    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8h7\" (UniqueName: \"kubernetes.io/projected/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-kube-api-access-fl8h7\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281139    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281173    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281186    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281260    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8586\" (UniqueName: \"kubernetes.io/projected/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-kube-api-access-g8586\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281275    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-konnectivity-ca\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281328    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281354    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281359    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-system-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281383    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-cnibin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281417    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e24b7db-f2a1-4500-9d82-963292cb7765-host\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281429    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65fab40a-d3b0-4bf1-9ab3-396dfccebe5f-konnectivity-ca\") pod \"konnectivity-agent-s8g6n\" (UID: \"65fab40a-d3b0-4bf1-9ab3-396dfccebe5f\") " pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281448    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-conf-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281499    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zfx\" (UniqueName: \"kubernetes.io/projected/500490ba-4b0f-4e40-9f8e-3906d62c458c-kube-api-access-f5zfx\") pod \"migrator-66df5d496b-rx6l8\" (UID: \"500490ba-4b0f-4e40-9f8e-3906d62c458c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281531    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281547    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281610    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281717    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-hostroot\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281737    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281770    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281790    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qs8\" (UniqueName: \"kubernetes.io/projected/5f6789dc-0c9b-4507-9615-93c3ec632af2-kube-api-access-t4qs8\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281833    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-snapshots\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-service-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281870    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72c64997-e326-411c-9f16-8ad1671ef849-signing-key\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281900    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-kubernetes\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281924    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-cni-binary-copy\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281933    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-snapshots\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281949    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-netns\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.281993    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282005    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.282045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282058    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-config\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282088    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-system-cni-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1090e22-f04c-4bda-b340-0966209254af-cni-binary-copy\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-crio-socket\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282171    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282213    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282244    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dmv\" (UniqueName: \"kubernetes.io/projected/d1090e22-f04c-4bda-b340-0966209254af-kube-api-access-h2dmv\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282213    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72c64997-e326-411c-9f16-8ad1671ef849-signing-key\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282279    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-config\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282281    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-slash\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282319    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282330    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282338    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72c64997-e326-411c-9f16-8ad1671ef849-signing-cabundle\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282354    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpwz\" (UniqueName: \"kubernetes.io/projected/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-kube-api-access-fgpwz\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282397    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tst\" (UniqueName: \"kubernetes.io/projected/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-kube-api-access-v8tst\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282415    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cnibin\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282404    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-service-ca-bundle\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282431    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282447    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-registration-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282479    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-run\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282498    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrjb\" (UniqueName: \"kubernetes.io/projected/7057aab3-480b-4bd1-81cc-e100e33b35d5-kube-api-access-8vrjb\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282510    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282515    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-kubelet\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282542    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-stats-auth\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282557    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72c64997-e326-411c-9f16-8ad1671ef849-signing-cabundle\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282604    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282690    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdnh\" (UniqueName: \"kubernetes.io/projected/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-api-access-wkdnh\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282726    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-var-lib-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282789    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-lib-modules\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282811    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvx2l\" (UniqueName: \"kubernetes.io/projected/7eb92403-d07f-4bc4-bf34-4a15adff4863-kube-api-access-fvx2l\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282823    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-stats-auth\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282835    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282940    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282967    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-socket-dir-parent\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.282996    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8kf\" (UniqueName: \"kubernetes.io/projected/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-kube-api-access-cb8kf\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.283031    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-modprobe-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.283046    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb92403-d07f-4bc4-bf34-4a15adff4863-cert\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.283061    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-systemd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.283079    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/6b213ff9-3e00-426a-901f-e5ca105d37e8-kube-api-access-pqdzc\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.283453 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.283353    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb92403-d07f-4bc4-bf34-4a15adff4863-cert\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:53.295799 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.295777    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbp2v\" (UniqueName: \"kubernetes.io/projected/b25a3f44-1814-43a8-a01d-f98c870b7d9f-kube-api-access-tbp2v\") pod \"volume-data-source-validator-58c9f5f46d-sf4kv\" (UID: \"b25a3f44-1814-43a8-a01d-f98c870b7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sf4kv"
Oct 30 23:53:53.296075 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.296057    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpsf\" (UniqueName: \"kubernetes.io/projected/6e24b7db-f2a1-4500-9d82-963292cb7765-kube-api-access-4jpsf\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.296358 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.296337    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.296434 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.296368    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4bc\" (UniqueName: \"kubernetes.io/projected/be5dad9f-114f-4b12-bd51-217374c94fc0-kube-api-access-nt4bc\") pod \"kas-connection-checker-685895c7c9-2qk9k\" (UID: \"be5dad9f-114f-4b12-bd51-217374c94fc0\") " pod="kube-system/kas-connection-checker-685895c7c9-2qk9k"
Oct 30 23:53:53.296434 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.296395    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86sq\" (UniqueName: \"kubernetes.io/projected/72c64997-e326-411c-9f16-8ad1671ef849-kube-api-access-q86sq\") pod \"service-ca-5fffc97bdf-84r2z\" (UID: \"72c64997-e326-411c-9f16-8ad1671ef849\") " pod="openshift-service-ca/service-ca-5fffc97bdf-84r2z"
Oct 30 23:53:53.296558 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.296456    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95p8n\" (UniqueName: \"kubernetes.io/projected/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-kube-api-access-95p8n\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.296558 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.296483    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltt9r\" (UniqueName: \"kubernetes.io/projected/88aedb21-8c5c-457b-a5c1-b778c33565c3-kube-api-access-ltt9r\") pod \"migrator-66df5d496b-fjlhr\" (UID: \"88aedb21-8c5c-457b-a5c1-b778c33565c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjlhr"
Oct 30 23:53:53.297378 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.297356    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4557\" (UniqueName: \"kubernetes.io/projected/fe484ab6-ac95-4849-acc0-37636302d300-kube-api-access-r4557\") pod \"network-check-source-6f78d59f99-rw5ms\" (UID: \"fe484ab6-ac95-4849-acc0-37636302d300\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rw5ms"
Oct 30 23:53:53.297647 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.297621    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.297745 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.297721    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwggr\" (UniqueName: \"kubernetes.io/projected/51bd9499-546b-4b4e-8350-7501c25b0ab2-kube-api-access-qwggr\") pod \"kas-connection-checker-685895c7c9-69tlv\" (UID: \"51bd9499-546b-4b4e-8350-7501c25b0ab2\") " pod="kube-system/kas-connection-checker-685895c7c9-69tlv"
Oct 30 23:53:53.298063 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.298046    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-kube-api-access-859wz\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.298442 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.298423    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhm77\" (UniqueName: \"kubernetes.io/projected/362b6ff8-9785-49c5-8704-db4c94a603d9-kube-api-access-mhm77\") pod \"network-check-target-2qs9j\" (UID: \"362b6ff8-9785-49c5-8704-db4c94a603d9\") " pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:53.298601 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.298586    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6w5v\" (UniqueName: \"kubernetes.io/projected/daf52a97-c6ca-4fba-a77b-67841614c162-kube-api-access-x6w5v\") pod \"service-ca-operator-544d749c8c-vqgls\" (UID: \"daf52a97-c6ca-4fba-a77b-67841614c162\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls"
Oct 30 23:53:53.299333 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.299319    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rft\" (UniqueName: \"kubernetes.io/projected/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-kube-api-access-k2rft\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.310589 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.310570    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8h7\" (UniqueName: \"kubernetes.io/projected/c4cac373-8436-4b03-a28b-0fa1bf8e7d13-kube-api-access-fl8h7\") pod \"insights-operator-7c4986d449-mb6hw\" (UID: \"c4cac373-8436-4b03-a28b-0fa1bf8e7d13\") " pod="openshift-insights/insights-operator-7c4986d449-mb6hw"
Oct 30 23:53:53.313686 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.313666    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmm2\" (UniqueName: \"kubernetes.io/projected/a286ab8f-3f4f-4621-8b10-3730b4e114f7-kube-api-access-dhmm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6\" (UID: \"a286ab8f-3f4f-4621-8b10-3730b4e114f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6"
Oct 30 23:53:53.313757 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.313704    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwq4\" (UniqueName: \"kubernetes.io/projected/25170a0c-24d2-48b8-80fa-d2d2780a41a3-kube-api-access-xkwq4\") pod \"kas-connection-checker-685895c7c9-kgs9w\" (UID: \"25170a0c-24d2-48b8-80fa-d2d2780a41a3\") " pod="kube-system/kas-connection-checker-685895c7c9-kgs9w"
Oct 30 23:53:53.314964 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.314948    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjsr\" (UniqueName: \"kubernetes.io/projected/964f2946-7176-4f03-b41f-03bd65aa8346-kube-api-access-pqjsr\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.316479 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.316449    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgh87\" (UniqueName: \"kubernetes.io/projected/3bbf2354-7b35-4374-adc3-e694b71f6e9e-kube-api-access-sgh87\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.317188 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.317172    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dmv\" (UniqueName: \"kubernetes.io/projected/d1090e22-f04c-4bda-b340-0966209254af-kube-api-access-h2dmv\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.319668 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.319648    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tst\" (UniqueName: \"kubernetes.io/projected/578a9d2b-c161-4a91-8f5a-8cf5a891a2bb-kube-api-access-v8tst\") pod \"console-operator-7cd56b8dbc-zzxg7\" (UID: \"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:53.320316 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.320296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdnh\" (UniqueName: \"kubernetes.io/projected/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-kube-api-access-wkdnh\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.320394 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.320360    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvx2l\" (UniqueName: \"kubernetes.io/projected/7eb92403-d07f-4bc4-bf34-4a15adff4863-kube-api-access-fvx2l\") pod \"ingress-canary-mrwcz\" (UID: \"7eb92403-d07f-4bc4-bf34-4a15adff4863\") " pod="openshift-ingress-canary/ingress-canary-mrwcz"
Oct 30 23:53:53.320529 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.320512    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zfx\" (UniqueName: \"kubernetes.io/projected/500490ba-4b0f-4e40-9f8e-3906d62c458c-kube-api-access-f5zfx\") pod \"migrator-66df5d496b-rx6l8\" (UID: \"500490ba-4b0f-4e40-9f8e-3906d62c458c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rx6l8"
Oct 30 23:53:53.321292 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.321272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrjb\" (UniqueName: \"kubernetes.io/projected/7057aab3-480b-4bd1-81cc-e100e33b35d5-kube-api-access-8vrjb\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.321534 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.321519    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8kf\" (UniqueName: \"kubernetes.io/projected/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-kube-api-access-cb8kf\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.321763 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.321746    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/6b213ff9-3e00-426a-901f-e5ca105d37e8-kube-api-access-pqdzc\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.321838 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.321820    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.321870 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.321832    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") pod \"periodic-gathering-rnrwh-lb8r2\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") " pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:53:53.321989 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.321972    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.322851 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.322828    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpwz\" (UniqueName: \"kubernetes.io/projected/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-kube-api-access-fgpwz\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.324786 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.324770    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qs8\" (UniqueName: \"kubernetes.io/projected/5f6789dc-0c9b-4507-9615-93c3ec632af2-kube-api-access-t4qs8\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.325584 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.325565    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8586\" (UniqueName: \"kubernetes.io/projected/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-kube-api-access-g8586\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.384241 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384219    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-conf\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.384241 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384242    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-node-log\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.384366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384256    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.384366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384270    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.384366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384292    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.384366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.384366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384345    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.384366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384348    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384345    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-node-log\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384383    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384403    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384412    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-kubelet-config\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384401    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-conf\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384406    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384480    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-dbus\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384493    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384617    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-kubelet-config\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384622    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-host\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384646    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e73a9af6-0664-409b-a096-83a990ec98ce-dbus\") pod \"global-pull-secret-syncer-ktssl\" (UID: \"e73a9af6-0664-409b-a096-83a990ec98ce\") " pod="kube-system/global-pull-secret-syncer-ktssl"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384656    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-host\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.384649 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384657    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-netd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384669    5078 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384682    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-netd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384680    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-kubelet\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384723    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384729    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-kubelet\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384745    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-bin\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384770    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384806    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-socket-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384821    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-log-socket\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384784    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384840    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384857    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384879    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-cni-bin\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384900    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-log-socket\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384883    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-socket-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384923    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-multus\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384952    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysctl-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-device-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.384993    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-k8s-cni-cncf-io\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385016    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-bin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385042    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-multus\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385037    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-device-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-k8s-cni-cncf-io\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385096    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-etc-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385041    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-etc-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385112    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-cni-bin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385168    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385204    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-multus-certs\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385219    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bbf2354-7b35-4374-adc3-e694b71f6e9e-hosts-file\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385238    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-systemd\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385243    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-multus-certs\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385254    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385274    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385293    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-system-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385297    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bbf2354-7b35-4374-adc3-e694b71f6e9e-hosts-file\") pod \"node-resolver-ccfzb\" (UID: \"3bbf2354-7b35-4374-adc3-e694b71f6e9e\") " pod="openshift-dns/node-resolver-ccfzb"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-cnibin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385327    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385352    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-systemd\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385360    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385379    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e24b7db-f2a1-4500-9d82-963292cb7765-host\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.385380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385406    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-cnibin\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385476    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:53.885437434 +0000 UTC m=+1.831808141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385345    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e24b7db-f2a1-4500-9d82-963292cb7765-host\") pod \"node-ca-bndnt\" (UID: \"6e24b7db-f2a1-4500-9d82-963292cb7765\") " pod="openshift-image-registry/node-ca-bndnt"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385444    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-system-cni-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385512    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-conf-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385543    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385567    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385598    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385594    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-hostroot\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385635    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385664    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-kubernetes\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385683    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-hostroot\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385712    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-netns\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385721    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-kubernetes\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385745    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-system-cni-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385772    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-crio-socket\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385781    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-run-netns\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385797    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385829    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-slash\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385831    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-system-cni-dir\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385829    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-crio-socket\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385775    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385862    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385861    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cnibin\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385889    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-registration-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385906    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:53.885890943 +0000 UTC m=+1.832261663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385920    5078 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385930    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-run\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385957    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-kubelet\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.385976    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:53.885954703 +0000 UTC m=+1.832325435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.385998    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-host-var-lib-kubelet\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386006    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-var-lib-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386032    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-lib-modules\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386058    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386083    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-socket-dir-parent\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386120    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-registration-dir\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386140    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-modprobe-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386170    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-systemd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386204    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-slash\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386193    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysconfig\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386162    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-run\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386253    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-ovn\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-sysconfig\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.386295    5078 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386291    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-sys-fs\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386315    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-var-lib-openvswitch\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.386336    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:53.886323902 +0000 UTC m=+1.832694599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386358    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-sys\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386375    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b213ff9-3e00-426a-901f-e5ca105d37e8-sys-fs\") pod \"aws-ebs-csi-driver-node-68c79\" (UID: \"6b213ff9-3e00-426a-901f-e5ca105d37e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386384    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-etc-kubernetes\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386411    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-ovn\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386410    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-host-slash\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386451    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa85bf8b-61bf-4bf1-b820-05215a7eadf3-host-slash\") pod \"iptables-alerter-jmmdd\" (UID: \"fa85bf8b-61bf-4bf1-b820-05215a7eadf3\") " pod="openshift-network-operator/iptables-alerter-jmmdd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386477    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-etc-modprobe-d\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386490    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-socket-dir-parent\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386527    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-etc-kubernetes\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386532    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-sys\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386548    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-lib-modules\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-run-systemd\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386587    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58-cnibin\") pod \"multus-additional-cni-plugins-7bbnp\" (UID: \"dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58\") " pod="openshift-multus/multus-additional-cni-plugins-7bbnp"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386602    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386644    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-var-lib-kubelet\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386740    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-systemd-units\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7057aab3-480b-4bd1-81cc-e100e33b35d5-var-lib-kubelet\") pod \"tuned-db4cd\" (UID: \"7057aab3-480b-4bd1-81cc-e100e33b35d5\") " pod="openshift-cluster-node-tuning-operator/tuned-db4cd"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386798    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-systemd-units\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386791    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-netns\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.386881    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a1a8190-b44f-41ac-8f0f-5020e6e27c18-host-run-netns\") pod \"ovnkube-node-hqmv6\" (UID: \"1a1a8190-b44f-41ac-8f0f-5020e6e27c18\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.387188    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:53.887171063 +0000 UTC m=+1.833541777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:53.387245 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.387194    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1090e22-f04c-4bda-b340-0966209254af-multus-conf-dir\") pod \"multus-4f6sj\" (UID: \"d1090e22-f04c-4bda-b340-0966209254af\") " pod="openshift-multus/multus-4f6sj"
Oct 30 23:53:53.389327 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.387485    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1384f8a-a2ad-4e55-b6e5-14ab134fa489-metrics-tls\") pod \"dns-default-shskm\" (UID: \"d1384f8a-a2ad-4e55-b6e5-14ab134fa489\") " pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.389327 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.387664    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b949609-5560-4b48-a5b7-229d7430d7c0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-k7rhq\" (UID: \"1b949609-5560-4b48-a5b7-229d7430d7c0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.389327 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.388083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6789dc-0c9b-4507-9615-93c3ec632af2-metrics-certs\") pod \"network-metrics-daemon-vl6sr\" (UID: \"5f6789dc-0c9b-4507-9615-93c3ec632af2\") " pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.389327 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.388119    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-metrics-certs\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.389327 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.389179    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964f2946-7176-4f03-b41f-03bd65aa8346-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h2j9x\" (UID: \"964f2946-7176-4f03-b41f-03bd65aa8346\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.454441 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.454422    5078 scope.go:122] "RemoveContainer" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160"
Oct 30 23:53:53.457673 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.457628    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"
Oct 30 23:53:53.459630 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.459616    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"
Oct 30 23:53:53.463894 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.463878    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vl6sr"
Oct 30 23:53:53.466697 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.466677    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:53.589648 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.589619    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:53:53.759828 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.759803    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-k7rhq"]
Oct 30 23:53:53.761400 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:53.761374    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b949609_5560_4b48_a5b7_229d7430d7c0.slice/crio-d34c8809c0a823e6b805353b525689dc98cfcd814305e2e9832f23f845f25c63 WatchSource:0}: Error finding container d34c8809c0a823e6b805353b525689dc98cfcd814305e2e9832f23f845f25c63: Status 404 returned error can't find the container with id d34c8809c0a823e6b805353b525689dc98cfcd814305e2e9832f23f845f25c63
Oct 30 23:53:53.763538 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.763520    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:53:53.766776 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.766756    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x"]
Oct 30 23:53:53.778529 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.778510    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vl6sr"]
Oct 30 23:53:53.781715 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:53.781688    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6789dc_0c9b_4507_9615_93c3ec632af2.slice/crio-c59e01aefb68fb048f810ad7696e5b26ea49a21c5374b71a1b567d0a6cbf52c8 WatchSource:0}: Error finding container c59e01aefb68fb048f810ad7696e5b26ea49a21c5374b71a1b567d0a6cbf52c8: Status 404 returned error can't find the container with id c59e01aefb68fb048f810ad7696e5b26ea49a21c5374b71a1b567d0a6cbf52c8
Oct 30 23:53:53.792338 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.792317    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-shskm"]
Oct 30 23:53:53.793912 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:53.793884    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1384f8a_a2ad_4e55_b6e5_14ab134fa489.slice/crio-4ba5fb2d10448a8b949af156b1617c20abb654e59be235b6c354551fbb60336c WatchSource:0}: Error finding container 4ba5fb2d10448a8b949af156b1617c20abb654e59be235b6c354551fbb60336c: Status 404 returned error can't find the container with id 4ba5fb2d10448a8b949af156b1617c20abb654e59be235b6c354551fbb60336c
Oct 30 23:53:53.891868 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.891835    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:53.892030 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.891892    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:53.892030 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.891982    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:53.892030 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.891994    5078 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:53.892030 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.892018    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:53.892045    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892067    5078 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892103    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:54.892084803 +0000 UTC m=+2.838455511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892145    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:54.89212572 +0000 UTC m=+2.838496432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892178    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892190    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892191    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:53.892233 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892210    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:53.892602 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892305    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:54.892172673 +0000 UTC m=+2.838543383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:53.892602 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892332    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:54.892317818 +0000 UTC m=+2.838688514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:53.892602 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:53.892347    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:54.892338958 +0000 UTC m=+2.838709652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:54.220395 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.220365    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s8g6n"
Oct 30 23:53:54.254256 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.254228    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:53:54.254395 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.254327    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" event={"ID":"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb","Type":"ContainerStarted","Data":"a6e89c3847eef8d6df2a2d3fd02f92c307a2828a4ca73c740c118e90d1ea7f0d"}
Oct 30 23:53:54.255516 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.255493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vl6sr" event={"ID":"5f6789dc-0c9b-4507-9615-93c3ec632af2","Type":"ContainerStarted","Data":"c59e01aefb68fb048f810ad7696e5b26ea49a21c5374b71a1b567d0a6cbf52c8"}
Oct 30 23:53:54.256670 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.256652    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x" event={"ID":"964f2946-7176-4f03-b41f-03bd65aa8346","Type":"ContainerStarted","Data":"19e573d2625ea741c264d0a53546712f630d9625cbbaf3ef0cff4a2bb0049c98"}
Oct 30 23:53:54.257885 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.257860    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-shskm" event={"ID":"d1384f8a-a2ad-4e55-b6e5-14ab134fa489","Type":"ContainerStarted","Data":"4ba5fb2d10448a8b949af156b1617c20abb654e59be235b6c354551fbb60336c"}
Oct 30 23:53:54.259004 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.258981    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq" event={"ID":"1b949609-5560-4b48-a5b7-229d7430d7c0","Type":"ContainerStarted","Data":"d34c8809c0a823e6b805353b525689dc98cfcd814305e2e9832f23f845f25c63"}
Oct 30 23:53:54.900751 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.900712    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:54.900950 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.900777    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:54.900950 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.900804    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:54.900950 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.900841    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:54.900950 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.900895    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:54.900950 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.900924    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:54.900883    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.900970    5078 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.900975    5078 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.900995    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.900965977 +0000 UTC m=+4.847336686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.901011    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.901020    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.901010908 +0000 UTC m=+4.847381602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.901021    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.901037    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.901029687 +0000 UTC m=+4.847400382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.901052    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls podName:a91f2be6-7bda-48a0-a0fc-8d5da8469dd7 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.90104538 +0000 UTC m=+4.847416075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l5ctp" (UID: "a91f2be6-7bda-48a0-a0fc-8d5da8469dd7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 23:53:54.901221 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:54.901068    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:56.901060966 +0000 UTC m=+4.847431660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:55.267404 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:55.267373    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-k7rhq" event={"ID":"1b949609-5560-4b48-a5b7-229d7430d7c0","Type":"ContainerStarted","Data":"9273d420cf6d7a4d9ca058ec3fe7d243c0b2d25509191fc8190eab8b98ec9e5c"}
Oct 30 23:53:56.249052 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.249025    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:56.249190 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.249134    5078 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:53:56.253233 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.253087    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 30 23:53:56.273072 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.273044    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vl6sr" event={"ID":"5f6789dc-0c9b-4507-9615-93c3ec632af2","Type":"ContainerStarted","Data":"2f4b82ea422ed633ec82ac4d10671d47b48c38fcdec0766fa3040e69980e9fc0"}
Oct 30 23:53:56.273414 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.273091    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vl6sr" event={"ID":"5f6789dc-0c9b-4507-9615-93c3ec632af2","Type":"ContainerStarted","Data":"f8ff14e46edb7a12adeb66b42425d14c24ad528152909ca83c519c994073f6af"}
Oct 30 23:53:56.274897 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.274876    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x" event={"ID":"964f2946-7176-4f03-b41f-03bd65aa8346","Type":"ContainerStarted","Data":"00b4f3f404fd175795882915338a2e720b67ec98fa24405b0707fb99787706e5"}
Oct 30 23:53:56.275009 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.274912    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h2j9x" event={"ID":"964f2946-7176-4f03-b41f-03bd65aa8346","Type":"ContainerStarted","Data":"87b406d14b020c8cb62a2bec8bd9c39c18616e8b78780464348e11750f96fc1d"}
Oct 30 23:53:56.277084 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.277027    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-shskm" event={"ID":"d1384f8a-a2ad-4e55-b6e5-14ab134fa489","Type":"ContainerStarted","Data":"d3c3aed3c44053afe1df55bbaf8adee916bdccdf3c5100e775ff31789cce18cd"}
Oct 30 23:53:56.277229 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.277122    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-shskm" event={"ID":"d1384f8a-a2ad-4e55-b6e5-14ab134fa489","Type":"ContainerStarted","Data":"75da5ce0713bbcce8f60c8a527d778c8aef1798ce6d0cba91a58981249f3ee86"}
Oct 30 23:53:56.277327 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.277285    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-shskm"
Oct 30 23:53:56.284416 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.284400    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:53:56.923993 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.923952    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:53:56.923993 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.923999    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.924020    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.924052    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.924093    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924124    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924141    5078 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924149    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449bcf677-rqxsk: secret "image-registry-tls" not found
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924155    5078 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674dd678bd-xdv8h: secret "image-registry-tls" not found
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924162    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle podName:d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:00.924140664 +0000 UTC m=+8.870511381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle") pod "router-default-74cc75bf99-fkb87" (UID: "d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1") : configmap references non-existent config key: service-ca.crt
Oct 30 23:53:56.924210 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924204    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls podName:36ab1743-a264-4439-a1a1-fd147ebd1248 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:00.924186538 +0000 UTC m=+8.870557248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls") pod "image-registry-674dd678bd-xdv8h" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248") : secret "image-registry-tls" not found
Oct 30 23:53:56.924649 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924252    5078 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:56.924649 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924261    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls podName:a5f9a68d-9d3a-4032-9f05-821e5563894a nodeName:}" failed. No retries permitted until 2026-10-30 23:54:00.924223748 +0000 UTC m=+8.870594443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls") pod "image-registry-6449bcf677-rqxsk" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a") : secret "image-registry-tls" not found
Oct 30 23:53:56.924649 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:53:56.924317    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls podName:567d58b9-e17a-41fd-a5e2-6ec9ff24597d nodeName:}" failed. No retries permitted until 2026-10-30 23:54:00.924299022 +0000 UTC m=+8.870669744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgtpq" (UID: "567d58b9-e17a-41fd-a5e2-6ec9ff24597d") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:53:56.926788 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:56.926764    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a91f2be6-7bda-48a0-a0fc-8d5da8469dd7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l5ctp\" (UID: \"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:57.058599 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:57.058525    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"
Oct 30 23:53:57.283366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:57.283333    5078 generic.go:356] "Generic (PLEG): container finished" podID="c4cac373-8436-4b03-a28b-0fa1bf8e7d13" containerID="87744ac71e54d0974de45a85708c2714ec65fabecf24afc706f14d7ce2fe7836" exitCode=0
Oct 30 23:53:57.283788 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:57.283400    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mb6hw" event={"ID":"c4cac373-8436-4b03-a28b-0fa1bf8e7d13","Type":"ContainerDied","Data":"87744ac71e54d0974de45a85708c2714ec65fabecf24afc706f14d7ce2fe7836"}
Oct 30 23:53:57.284145 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:57.284128    5078 scope.go:122] "RemoveContainer" containerID="87744ac71e54d0974de45a85708c2714ec65fabecf24afc706f14d7ce2fe7836"
Oct 30 23:53:57.337103 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:57.337084    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp"]
Oct 30 23:53:57.352045 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:53:57.352019    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91f2be6_7bda_48a0_a0fc_8d5da8469dd7.slice/crio-cf5a246bf7e9243c8bca70fd6176ca3b05f6d7617a177a3571603776665cb52d WatchSource:0}: Error finding container cf5a246bf7e9243c8bca70fd6176ca3b05f6d7617a177a3571603776665cb52d: Status 404 returned error can't find the container with id cf5a246bf7e9243c8bca70fd6176ca3b05f6d7617a177a3571603776665cb52d
Oct 30 23:53:58.291117 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:58.291080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mb6hw" event={"ID":"c4cac373-8436-4b03-a28b-0fa1bf8e7d13","Type":"ContainerStarted","Data":"6dce8c54d293b831adf9e571f670c6b8e784a96e8cf3950b39cf1a1c55e2a75b"}
Oct 30 23:53:58.293240 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:58.293210    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6" event={"ID":"a286ab8f-3f4f-4621-8b10-3730b4e114f7","Type":"ContainerDied","Data":"4e177a19e18d57461800e58fdb65b7ccdcb0205328568ca5234a9b55a98412f9"}
Oct 30 23:53:58.293358 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:58.293129    5078 generic.go:356] "Generic (PLEG): container finished" podID="a286ab8f-3f4f-4621-8b10-3730b4e114f7" containerID="4e177a19e18d57461800e58fdb65b7ccdcb0205328568ca5234a9b55a98412f9" exitCode=0
Oct 30 23:53:58.293537 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:58.293523    5078 scope.go:122] "RemoveContainer" containerID="4e177a19e18d57461800e58fdb65b7ccdcb0205328568ca5234a9b55a98412f9"
Oct 30 23:53:58.295146 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:58.295123    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp" event={"ID":"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7","Type":"ContainerStarted","Data":"cf5a246bf7e9243c8bca70fd6176ca3b05f6d7617a177a3571603776665cb52d"}
Oct 30 23:53:59.305835 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:59.305801    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qrpc6" event={"ID":"a286ab8f-3f4f-4621-8b10-3730b4e114f7","Type":"ContainerStarted","Data":"cd03765fe37808d6ff01e31479575f3d5bee62ef249fb2b1cd0b9bd6d725bb4e"}
Oct 30 23:53:59.308370 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:53:59.308340    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l5ctp" event={"ID":"a91f2be6-7bda-48a0-a0fc-8d5da8469dd7","Type":"ContainerStarted","Data":"b9250edf3a962002651718a7e785d67510c42b49da63490a944d82cfbbba2e4f"}
Oct 30 23:54:00.964181 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.964142    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:54:00.964635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.964243    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:00.964635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.964276    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:00.964635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.964291    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:00.965029 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.965003    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1-service-ca-bundle\") pod \"router-default-74cc75bf99-fkb87\" (UID: \"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1\") " pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:00.966856 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.966831    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/567d58b9-e17a-41fd-a5e2-6ec9ff24597d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgtpq\" (UID: \"567d58b9-e17a-41fd-a5e2-6ec9ff24597d\") " pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:54:00.966954 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.966838    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"image-registry-674dd678bd-xdv8h\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") " pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:00.966954 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.966941    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"image-registry-6449bcf677-rqxsk\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") " pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:00.972710 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:00.972690    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgtpq"
Oct 30 23:54:01.248936 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.248841    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgtpq"]
Oct 30 23:54:01.251792 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:01.251756    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567d58b9_e17a_41fd_a5e2_6ec9ff24597d.slice/crio-6cb4aa039492bdbbfaf505c2d4653f46b80b48cb3f71f28bef7215be0fcf7111 WatchSource:0}: Error finding container 6cb4aa039492bdbbfaf505c2d4653f46b80b48cb3f71f28bef7215be0fcf7111: Status 404 returned error can't find the container with id 6cb4aa039492bdbbfaf505c2d4653f46b80b48cb3f71f28bef7215be0fcf7111
Oct 30 23:54:01.258603 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.258583    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:01.259671 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.259649    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:01.265810 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.265787    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:01.319273 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.319245    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgtpq" event={"ID":"567d58b9-e17a-41fd-a5e2-6ec9ff24597d","Type":"ContainerStarted","Data":"6cb4aa039492bdbbfaf505c2d4653f46b80b48cb3f71f28bef7215be0fcf7111"}
Oct 30 23:54:01.554737 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.554676    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74cc75bf99-fkb87"]
Oct 30 23:54:01.558040 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.558016    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674dd678bd-xdv8h"]
Oct 30 23:54:01.558181 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:01.558141    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e8e526_6b9d_4aff_9bd3_3b38c1461aa1.slice/crio-8056d5fa6e8dbb19c23d515f03153fc75eb05402248b349b9d4a0a99e998cab1 WatchSource:0}: Error finding container 8056d5fa6e8dbb19c23d515f03153fc75eb05402248b349b9d4a0a99e998cab1: Status 404 returned error can't find the container with id 8056d5fa6e8dbb19c23d515f03153fc75eb05402248b349b9d4a0a99e998cab1
Oct 30 23:54:01.560728 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:01.560667    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ab1743_a264_4439_a1a1_fd147ebd1248.slice/crio-17bc437717fe63681ea75c80369b59cd2216cf9d52862c35a76d534c0fecf2db WatchSource:0}: Error finding container 17bc437717fe63681ea75c80369b59cd2216cf9d52862c35a76d534c0fecf2db: Status 404 returned error can't find the container with id 17bc437717fe63681ea75c80369b59cd2216cf9d52862c35a76d534c0fecf2db
Oct 30 23:54:01.561058 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:01.560992    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6449bcf677-rqxsk"]
Oct 30 23:54:01.563578 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:01.563558    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f9a68d_9d3a_4032_9f05_821e5563894a.slice/crio-81a66359229db7b1b5aa1c13ee057278e42320b6d39b6ca1ddbd9e09905a5a9e WatchSource:0}: Error finding container 81a66359229db7b1b5aa1c13ee057278e42320b6d39b6ca1ddbd9e09905a5a9e: Status 404 returned error can't find the container with id 81a66359229db7b1b5aa1c13ee057278e42320b6d39b6ca1ddbd9e09905a5a9e
Oct 30 23:54:02.324949 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.324856    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74cc75bf99-fkb87" event={"ID":"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1","Type":"ContainerStarted","Data":"6a8f117cb1730c3a9d8dd12471c1758fe1e91086c7d36273454a6d9e1a23fd4b"}
Oct 30 23:54:02.324949 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.324922    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74cc75bf99-fkb87" event={"ID":"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1","Type":"ContainerStarted","Data":"8056d5fa6e8dbb19c23d515f03153fc75eb05402248b349b9d4a0a99e998cab1"}
Oct 30 23:54:02.326965 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.326943    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgtpq" event={"ID":"567d58b9-e17a-41fd-a5e2-6ec9ff24597d","Type":"ContainerStarted","Data":"4f8a335ff6a5dd31ad898e3fb4b03b910be783c821515e424771f26ffde33501"}
Oct 30 23:54:02.327097 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.326984    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgtpq" event={"ID":"567d58b9-e17a-41fd-a5e2-6ec9ff24597d","Type":"ContainerStarted","Data":"e6991cb10343c376568e5b43dbd0500f603e4157c6650d6421be6c77e0d860bd"}
Oct 30 23:54:02.328354 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.328331    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk" event={"ID":"a5f9a68d-9d3a-4032-9f05-821e5563894a","Type":"ContainerStarted","Data":"9f9dedd71ade7e81904888f04bbfc66b5c7ddc82b6e64f18ad4328ce0574c9eb"}
Oct 30 23:54:02.328482 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.328374    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk" event={"ID":"a5f9a68d-9d3a-4032-9f05-821e5563894a","Type":"ContainerStarted","Data":"81a66359229db7b1b5aa1c13ee057278e42320b6d39b6ca1ddbd9e09905a5a9e"}
Oct 30 23:54:02.328482 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.328431    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:02.330104 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.330079    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h" event={"ID":"36ab1743-a264-4439-a1a1-fd147ebd1248","Type":"ContainerStarted","Data":"716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7"}
Oct 30 23:54:02.330202 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.330124    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h" event={"ID":"36ab1743-a264-4439-a1a1-fd147ebd1248","Type":"ContainerStarted","Data":"17bc437717fe63681ea75c80369b59cd2216cf9d52862c35a76d534c0fecf2db"}
Oct 30 23:54:02.558482 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.558425    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:54:02.558683 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.558670    5078 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:54:02.816821 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:02.816787    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6"
Oct 30 23:54:03.259540 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:03.259507    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:03.262563 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:03.262452    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:03.334784 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:03.334758    5078 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:54:04.344837 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:04.344802    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgtpq" event={"ID":"567d58b9-e17a-41fd-a5e2-6ec9ff24597d","Type":"ContainerStarted","Data":"cda4b8e36915b19fe3c07f3d227da0f3da6829f665db096554ceedf6d41b8c74"}
Oct 30 23:54:04.345338 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:04.344869    5078 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:54:05.025315 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:05.025284    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:06.285789 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:06.285764    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-shskm"
Oct 30 23:54:06.341702 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:06.341672    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:06.342015 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:06.341798    5078 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:54:06.342940 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:06.342917    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 30 23:54:06.385551 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:54:06.385516    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:54:07.784486 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.784422    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gt7cj-g4r94"]
Oct 30 23:54:07.790835 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.790812    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:07.808605 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.808579    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gt7cj-g4r94"]
Oct 30 23:54:07.821244 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.820806    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674dd678bd-xdv8h"]
Oct 30 23:54:07.832377 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.832329    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6wdkk"]
Oct 30 23:54:07.856326 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.856290    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"]
Oct 30 23:54:07.857167 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.857138    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:07.860253 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.860233    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6wdkk"]
Oct 30 23:54:07.860371 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.860269    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"]
Oct 30 23:54:07.860371 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.860364    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:07.872216 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.872193    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:54:07.877773 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.875525    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 23:54:07.877773 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.875643    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-dm6n2\""
Oct 30 23:54:07.877773 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.875731    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bzdk4\""
Oct 30 23:54:07.877773 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.875885    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:54:07.916593 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.916565    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c895f9d65-tkw4d"]
Oct 30 23:54:07.920068 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.920044    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:07.925650 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.925621    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:07.925875 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.925855    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:07.925995 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.925896    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7xw\" (UniqueName: \"kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:07.925995 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.925913    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:07.942406 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:07.942384    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c895f9d65-tkw4d"]
Oct 30 23:54:08.026584 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026553    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d46ade2d-3c27-496c-b4c4-d0a80f55f9a0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-94hrt\" (UID: \"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:08.026749 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026598    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f6a88c2-1453-4033-8027-afc8b69d3ee7-trusted-ca\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.026749 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026623    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-bound-sa-token\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.026749 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026702    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cxn\" (UniqueName: \"kubernetes.io/projected/b6209fdf-df1c-4a4f-b5ce-55812e9f07a4-kube-api-access-p5cxn\") pod \"downloads-6dd789c8b7-6wdkk\" (UID: \"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4\") " pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.026920 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026763    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f6a88c2-1453-4033-8027-afc8b69d3ee7-installation-pull-secrets\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.026920 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026790    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbcm\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-kube-api-access-7cbcm\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.026920 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026808    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6209fdf-df1c-4a4f-b5ce-55812e9f07a4-tmp\") pod \"downloads-6dd789c8b7-6wdkk\" (UID: \"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4\") " pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.026920 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.027112 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026922    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f6a88c2-1453-4033-8027-afc8b69d3ee7-registry-certificates\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.027112 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.026939    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-registry-tls\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.027112 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027003    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f6a88c2-1453-4033-8027-afc8b69d3ee7-image-registry-private-configuration\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.027112 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027047    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f6a88c2-1453-4033-8027-afc8b69d3ee7-ca-trust-extracted\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.027112 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027085    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.027286 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027172    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7xw\" (UniqueName: \"kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.027286 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027187    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.027286 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027215    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.027728 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.027707    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.028159 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.028131    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.044854 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.044796    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7xw\" (UniqueName: \"kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw\") pod \"periodic-gathering-gt7cj-g4r94\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") " pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.104852 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.103628    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:54:08.128059 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128036    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f6a88c2-1453-4033-8027-afc8b69d3ee7-registry-certificates\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128186 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128070    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-registry-tls\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128186 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128101    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f6a88c2-1453-4033-8027-afc8b69d3ee7-image-registry-private-configuration\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128186 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128130    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f6a88c2-1453-4033-8027-afc8b69d3ee7-ca-trust-extracted\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128355 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128204    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d46ade2d-3c27-496c-b4c4-d0a80f55f9a0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-94hrt\" (UID: \"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:08.128355 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f6a88c2-1453-4033-8027-afc8b69d3ee7-trusted-ca\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128355 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128255    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-bound-sa-token\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128355 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128284    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cxn\" (UniqueName: \"kubernetes.io/projected/b6209fdf-df1c-4a4f-b5ce-55812e9f07a4-kube-api-access-p5cxn\") pod \"downloads-6dd789c8b7-6wdkk\" (UID: \"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4\") " pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.128355 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128331    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f6a88c2-1453-4033-8027-afc8b69d3ee7-installation-pull-secrets\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128628 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128362    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbcm\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-kube-api-access-7cbcm\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.128628 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128407    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6209fdf-df1c-4a4f-b5ce-55812e9f07a4-tmp\") pod \"downloads-6dd789c8b7-6wdkk\" (UID: \"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4\") " pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.128853 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.128829    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6209fdf-df1c-4a4f-b5ce-55812e9f07a4-tmp\") pod \"downloads-6dd789c8b7-6wdkk\" (UID: \"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4\") " pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.129091 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.129067    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f6a88c2-1453-4033-8027-afc8b69d3ee7-ca-trust-extracted\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.129299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.129269    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f6a88c2-1453-4033-8027-afc8b69d3ee7-trusted-ca\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.130536 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.130510    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f6a88c2-1453-4033-8027-afc8b69d3ee7-registry-certificates\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.131854 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.131835    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-registry-tls\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.133137 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.133111    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d46ade2d-3c27-496c-b4c4-d0a80f55f9a0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-94hrt\" (UID: \"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:08.135837 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.135816    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f6a88c2-1453-4033-8027-afc8b69d3ee7-image-registry-private-configuration\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.136381 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.136361    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f6a88c2-1453-4033-8027-afc8b69d3ee7-installation-pull-secrets\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.142605 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.142584    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-bound-sa-token\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.143671 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.143642    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbcm\" (UniqueName: \"kubernetes.io/projected/2f6a88c2-1453-4033-8027-afc8b69d3ee7-kube-api-access-7cbcm\") pod \"image-registry-6c895f9d65-tkw4d\" (UID: \"2f6a88c2-1453-4033-8027-afc8b69d3ee7\") " pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.146563 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.146539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cxn\" (UniqueName: \"kubernetes.io/projected/b6209fdf-df1c-4a4f-b5ce-55812e9f07a4-kube-api-access-p5cxn\") pod \"downloads-6dd789c8b7-6wdkk\" (UID: \"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4\") " pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.181832 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.181803    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:08.190062 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.190036    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:08.234681 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.234648    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:08.400599 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.399990    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gt7cj-g4r94"]
Oct 30 23:54:08.501831 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.501803    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6wdkk"]
Oct 30 23:54:08.518422 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:08.518372    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6209fdf_df1c_4a4f_b5ce_55812e9f07a4.slice/crio-810e2ee5bf3ead13dd6791ee289223645a720d3cedf6b6f82a4e5480a3cf1aa2 WatchSource:0}: Error finding container 810e2ee5bf3ead13dd6791ee289223645a720d3cedf6b6f82a4e5480a3cf1aa2: Status 404 returned error can't find the container with id 810e2ee5bf3ead13dd6791ee289223645a720d3cedf6b6f82a4e5480a3cf1aa2
Oct 30 23:54:08.528011 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.527981    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"]
Oct 30 23:54:08.537548 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:08.537522    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46ade2d_3c27_496c_b4c4_d0a80f55f9a0.slice/crio-c5adfd431611e2b0ddd765036602c576d1e0dc0e1fae97ddcaac7cab221a71af WatchSource:0}: Error finding container c5adfd431611e2b0ddd765036602c576d1e0dc0e1fae97ddcaac7cab221a71af: Status 404 returned error can't find the container with id c5adfd431611e2b0ddd765036602c576d1e0dc0e1fae97ddcaac7cab221a71af
Oct 30 23:54:08.561748 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:08.561665    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c895f9d65-tkw4d"]
Oct 30 23:54:09.373536 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.373500    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" event={"ID":"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0","Type":"ContainerStarted","Data":"c5adfd431611e2b0ddd765036602c576d1e0dc0e1fae97ddcaac7cab221a71af"}
Oct 30 23:54:09.375534 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.375511    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" event={"ID":"2f6a88c2-1453-4033-8027-afc8b69d3ee7","Type":"ContainerStarted","Data":"94d6f2c9a258f8c20a0a432e9e3a78821fef99fd277f1cd49560a5cc99d0448e"}
Oct 30 23:54:09.375671 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.375546    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" event={"ID":"2f6a88c2-1453-4033-8027-afc8b69d3ee7","Type":"ContainerStarted","Data":"2b37a2163b61905cc35f149de6a36fb2001b47b1108d965a05185e8e491e82cd"}
Oct 30 23:54:09.375671 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.375657    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:09.377269 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.377246    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6wdkk" event={"ID":"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4","Type":"ContainerStarted","Data":"810e2ee5bf3ead13dd6791ee289223645a720d3cedf6b6f82a4e5480a3cf1aa2"}
Oct 30 23:54:09.379747 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.379679    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gt7cj-g4r94" event={"ID":"967c5ac1-949a-4f15-94d6-a4fa8342ee73","Type":"ContainerStarted","Data":"7c66c2af5c59dd88127447f1546e95ae5de21070092a1478279ec9b15248d61d"}
Oct 30 23:54:09.379747 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.379717    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gt7cj-g4r94" event={"ID":"967c5ac1-949a-4f15-94d6-a4fa8342ee73","Type":"ContainerStarted","Data":"163602b167787b3f61ed6f274db6e2aa5e5206024300b68f7d2f5d0f40abc288"}
Oct 30 23:54:09.408158 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.408107    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" podStartSLOduration=2.408091501 podStartE2EDuration="2.408091501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:09.405602958 +0000 UTC m=+17.351973705" watchObservedRunningTime="2026-10-30 23:54:09.408091501 +0000 UTC m=+17.354462216"
Oct 30 23:54:09.428775 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:09.428068    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gt7cj-g4r94" podStartSLOduration=2.42805551 podStartE2EDuration="2.42805551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:09.427439749 +0000 UTC m=+17.373810468" watchObservedRunningTime="2026-10-30 23:54:09.42805551 +0000 UTC m=+17.374426245"
Oct 30 23:54:10.127338 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.127265    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-shskm_d1384f8a-a2ad-4e55-b6e5-14ab134fa489/dns/0.log"
Oct 30 23:54:10.328282 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.328255    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-shskm_d1384f8a-a2ad-4e55-b6e5-14ab134fa489/kube-rbac-proxy/0.log"
Oct 30 23:54:10.387180 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.387115    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" event={"ID":"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0","Type":"ContainerStarted","Data":"b74c99cff5553b203610ed6531ef7db49fc1ff64817a62ae04ea5be20bd7ce64"}
Oct 30 23:54:10.387635 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.387208    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:10.392759 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.392737    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 30 23:54:10.408218 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.408169    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podStartSLOduration=2.262812986 podStartE2EDuration="3.40815209s" totalImagesPullingTime="1.145339104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:10.406852104 +0000 UTC m=+18.353222847" watchObservedRunningTime="2026-10-30 23:54:10.40815209 +0000 UTC m=+18.354522809"
Oct 30 23:54:10.527526 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:10.527499    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ccfzb_3bbf2354-7b35-4374-adc3-e694b71f6e9e/dns-node-resolver/0.log"
Oct 30 23:54:11.127656 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:11.127626    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6449bcf677-rqxsk_a5f9a68d-9d3a-4032-9f05-821e5563894a/registry/0.log"
Oct 30 23:54:11.328689 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:11.328656    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-674dd678bd-xdv8h_36ab1743-a264-4439-a1a1-fd147ebd1248/registry/0.log"
Oct 30 23:54:11.528939 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:11.528909    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c895f9d65-tkw4d_2f6a88c2-1453-4033-8027-afc8b69d3ee7/registry/0.log"
Oct 30 23:54:11.928985 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:11.928956    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bndnt_6e24b7db-f2a1-4500-9d82-963292cb7765/node-ca/0.log"
Oct 30 23:54:12.327832 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:12.327729    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 30 23:54:12.931177 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:12.931149    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrwcz_7eb92403-d07f-4bc4-bf34-4a15adff4863/serve-healthcheck-canary/0.log"
Oct 30 23:54:16.633655 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.632576    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kpj57"]
Oct 30 23:54:16.639629 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.639601    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.699553 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.699518    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 23:54:16.699553 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.699543    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 23:54:16.708509 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.708487    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 23:54:16.709005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.708980    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9slr\""
Oct 30 23:54:16.709107 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.709019    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 23:54:16.808302 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808264    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-wtmp\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808503 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808336    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c25f317f-5d84-42c6-b06d-e69dbb04420e-metrics-client-ca\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808503 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808446    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-root\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808609 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808514    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-textfile\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808609 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808538    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-accelerators-collector-config\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808609 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808562    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lwk\" (UniqueName: \"kubernetes.io/projected/c25f317f-5d84-42c6-b06d-e69dbb04420e-kube-api-access-97lwk\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808764 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808683    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-sys\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808764 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808718    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.808863 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.808800    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-tls\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909708 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909666    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-textfile\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909708 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909714    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-accelerators-collector-config\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909736    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lwk\" (UniqueName: \"kubernetes.io/projected/c25f317f-5d84-42c6-b06d-e69dbb04420e-kube-api-access-97lwk\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909763    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-sys\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909788    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909823    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-tls\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909863    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-wtmp\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909913    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c25f317f-5d84-42c6-b06d-e69dbb04420e-metrics-client-ca\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.909963 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.909938    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-root\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.910301 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.910031    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-root\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.910301 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.910083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-sys\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.910301 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.910084    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-textfile\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.910301 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.910203    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-wtmp\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.910524 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.910363    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-accelerators-collector-config\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.910633 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.910616    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c25f317f-5d84-42c6-b06d-e69dbb04420e-metrics-client-ca\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.912681 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.912638    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.912835 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.912817    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c25f317f-5d84-42c6-b06d-e69dbb04420e-node-exporter-tls\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.943550 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.943524    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lwk\" (UniqueName: \"kubernetes.io/projected/c25f317f-5d84-42c6-b06d-e69dbb04420e-kube-api-access-97lwk\") pod \"node-exporter-kpj57\" (UID: \"c25f317f-5d84-42c6-b06d-e69dbb04420e\") " pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.960196 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:16.960166    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kpj57"
Oct 30 23:54:16.976154 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:54:16.976119    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25f317f_5d84_42c6_b06d_e69dbb04420e.slice/crio-eff6d72ff7bd78d3e9d56b80ddd3e785667cb94d9b04af6ef77e5363533dab4e WatchSource:0}: Error finding container eff6d72ff7bd78d3e9d56b80ddd3e785667cb94d9b04af6ef77e5363533dab4e: Status 404 returned error can't find the container with id eff6d72ff7bd78d3e9d56b80ddd3e785667cb94d9b04af6ef77e5363533dab4e
Oct 30 23:54:17.426376 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:17.426343    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpj57" event={"ID":"c25f317f-5d84-42c6-b06d-e69dbb04420e","Type":"ContainerStarted","Data":"eff6d72ff7bd78d3e9d56b80ddd3e785667cb94d9b04af6ef77e5363533dab4e"}
Oct 30 23:54:17.826016 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:17.825933    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:18.433624 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:18.433583    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpj57" event={"ID":"c25f317f-5d84-42c6-b06d-e69dbb04420e","Type":"ContainerStarted","Data":"995d6b8530811c3850bfd7cf35c47f6f48fc4bf68ce8a7c5474f0d7cfcf0a4b5"}
Oct 30 23:54:19.441819 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:19.441782    5078 generic.go:356] "Generic (PLEG): container finished" podID="c25f317f-5d84-42c6-b06d-e69dbb04420e" containerID="995d6b8530811c3850bfd7cf35c47f6f48fc4bf68ce8a7c5474f0d7cfcf0a4b5" exitCode=0
Oct 30 23:54:19.442248 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:19.441897    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpj57" event={"ID":"c25f317f-5d84-42c6-b06d-e69dbb04420e","Type":"ContainerDied","Data":"995d6b8530811c3850bfd7cf35c47f6f48fc4bf68ce8a7c5474f0d7cfcf0a4b5"}
Oct 30 23:54:20.449580 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:20.449539    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpj57" event={"ID":"c25f317f-5d84-42c6-b06d-e69dbb04420e","Type":"ContainerStarted","Data":"2cd6e15bd770a490dd5cea6fe0ff12c36c367bf6fdebb64e30d28586a97cc2b7"}
Oct 30 23:54:20.450113 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:20.449605    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpj57" event={"ID":"c25f317f-5d84-42c6-b06d-e69dbb04420e","Type":"ContainerStarted","Data":"daf2974b13dd03315eb6c99bb907a049b7c780e3e4816b6487fc1d4429a0d071"}
Oct 30 23:54:20.479397 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:20.479348    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kpj57" podStartSLOduration=3.057740749 podStartE2EDuration="4.479327109s" totalImagesPullingTime="1.336163513s" totalInitContainerRuntime="85.422847ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:20.476067879 +0000 UTC m=+28.422438622" watchObservedRunningTime="2026-10-30 23:54:20.479327109 +0000 UTC m=+28.425697827"
Oct 30 23:54:23.340898 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:23.340859    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:25.478226 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:25.478143    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6wdkk" event={"ID":"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4","Type":"ContainerStarted","Data":"f210307b6ed0f7d4f6c22410102133a604ac8a14823f1c02613cdcb331ca370d"}
Oct 30 23:54:25.478689 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:25.478376    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:25.480232 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:25.480199    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Oct 30 23:54:25.480364 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:25.480253    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Oct 30 23:54:25.513063 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:25.513018    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podStartSLOduration=1.8815209560000001 podStartE2EDuration="18.513005872s" totalImagesPullingTime="16.631484916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:25.504914593 +0000 UTC m=+33.451285312" watchObservedRunningTime="2026-10-30 23:54:25.513005872 +0000 UTC m=+33.459376589"
Oct 30 23:54:26.287974 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:26.287938    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2qs9j"
Oct 30 23:54:26.510321 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:26.510287    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 30 23:54:30.397551 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:30.396565    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d"
Oct 30 23:54:30.474529 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:30.471034    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6449bcf677-rqxsk"]
Oct 30 23:54:32.888714 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:32.888587    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h" podUID="36ab1743-a264-4439-a1a1-fd147ebd1248" containerName="registry" containerID="cri-o://716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7" gracePeriod=30
Oct 30 23:54:33.187669 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.187641    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:33.280832 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.280799    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281036 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.280854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281036 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.280895    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281036 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.280978    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281036 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281020    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281263 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281060    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281263 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281109    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281263 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") pod \"36ab1743-a264-4439-a1a1-fd147ebd1248\" (UID: \"36ab1743-a264-4439-a1a1-fd147ebd1248\") "
Oct 30 23:54:33.281414 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281321    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:33.281502 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281443    5078 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-certificates\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.282013 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.281961    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:33.284205 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.284172    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:33.284692 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.284644    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:33.285380 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.285134    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "kube-api-access-npz6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:33.285870 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.285843    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:33.292433 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.292215    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:33.294904 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.294882    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets" pod "36ab1743-a264-4439-a1a1-fd147ebd1248" (UID: "36ab1743-a264-4439-a1a1-fd147ebd1248"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:33.382638 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382604    5078 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36ab1743-a264-4439-a1a1-fd147ebd1248-trusted-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.382638 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382632    5078 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-bound-sa-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.382826 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382649    5078 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-image-registry-private-configuration\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.382826 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382664    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz6x\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-kube-api-access-npz6x\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.382826 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382676    5078 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36ab1743-a264-4439-a1a1-fd147ebd1248-registry-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.382826 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382684    5078 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36ab1743-a264-4439-a1a1-fd147ebd1248-ca-trust-extracted\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.382826 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.382692    5078 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36ab1743-a264-4439-a1a1-fd147ebd1248-installation-pull-secrets\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:33.528038 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.527963    5078 generic.go:356] "Generic (PLEG): container finished" podID="36ab1743-a264-4439-a1a1-fd147ebd1248" containerID="716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7" exitCode=0
Oct 30 23:54:33.528038 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.528024    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h"
Oct 30 23:54:33.528247 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.528051    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h" event={"ID":"36ab1743-a264-4439-a1a1-fd147ebd1248","Type":"ContainerDied","Data":"716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7"}
Oct 30 23:54:33.528247 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.528115    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674dd678bd-xdv8h" event={"ID":"36ab1743-a264-4439-a1a1-fd147ebd1248","Type":"ContainerDied","Data":"17bc437717fe63681ea75c80369b59cd2216cf9d52862c35a76d534c0fecf2db"}
Oct 30 23:54:33.528247 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.528144    5078 scope.go:122] "RemoveContainer" containerID="716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7"
Oct 30 23:54:33.552742 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.552717    5078 scope.go:122] "RemoveContainer" containerID="716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7"
Oct 30 23:54:33.553158 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:54:33.553128    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7\": container with ID starting with 716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7 not found: ID does not exist" containerID="716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7"
Oct 30 23:54:33.553266 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.553169    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7"} err="failed to get container status \"716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7\": rpc error: code = NotFound desc = could not find container \"716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7\": container with ID starting with 716be46546dfaafdd727a2c1597988845ac8b69ca15dfb11133e1bf6a9b5f2d7 not found: ID does not exist"
Oct 30 23:54:33.556739 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.556720    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674dd678bd-xdv8h"]
Oct 30 23:54:33.562090 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:33.562064    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-674dd678bd-xdv8h"]
Oct 30 23:54:34.192671 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:34.192640    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ab1743-a264-4439-a1a1-fd147ebd1248" path="/var/lib/kubelet/pods/36ab1743-a264-4439-a1a1-fd147ebd1248/volumes"
Oct 30 23:54:36.384188 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:54:36.384154    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:54:39.306361 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.306322    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rnrwh-lb8r2"]
Oct 30 23:54:39.306846 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.306589    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2" podUID="4498189b-b2cb-46e0-b20f-926be273f49a" containerName="insights-gathering" containerID="cri-o://b2207b815352754bc0a3c1df853d799b6a2f40fe5577eff1e8e70d4f817d4408" gracePeriod=30
Oct 30 23:54:39.559483 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.559400    5078 generic.go:356] "Generic (PLEG): container finished" podID="4498189b-b2cb-46e0-b20f-926be273f49a" containerID="b2207b815352754bc0a3c1df853d799b6a2f40fe5577eff1e8e70d4f817d4408" exitCode=2
Oct 30 23:54:39.559483 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.559447    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2" event={"ID":"4498189b-b2cb-46e0-b20f-926be273f49a","Type":"ContainerDied","Data":"b2207b815352754bc0a3c1df853d799b6a2f40fe5577eff1e8e70d4f817d4408"}
Oct 30 23:54:39.624224 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.624200    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:54:39.732217 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732193    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") pod \"4498189b-b2cb-46e0-b20f-926be273f49a\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") "
Oct 30 23:54:39.732366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") pod \"4498189b-b2cb-46e0-b20f-926be273f49a\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") "
Oct 30 23:54:39.732366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732297    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") pod \"4498189b-b2cb-46e0-b20f-926be273f49a\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") "
Oct 30 23:54:39.732366 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732335    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") pod \"4498189b-b2cb-46e0-b20f-926be273f49a\" (UID: \"4498189b-b2cb-46e0-b20f-926be273f49a\") "
Oct 30 23:54:39.732611 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732582    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path" pod "4498189b-b2cb-46e0-b20f-926be273f49a" (UID: "4498189b-b2cb-46e0-b20f-926be273f49a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:39.732712 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732604    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle" pod "4498189b-b2cb-46e0-b20f-926be273f49a" (UID: "4498189b-b2cb-46e0-b20f-926be273f49a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:39.732789 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.732768    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle" pod "4498189b-b2cb-46e0-b20f-926be273f49a" (UID: "4498189b-b2cb-46e0-b20f-926be273f49a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:39.734899 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.734877    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f" pod "4498189b-b2cb-46e0-b20f-926be273f49a" (UID: "4498189b-b2cb-46e0-b20f-926be273f49a"). InnerVolumeSpecName "kube-api-access-zb26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:39.833758 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.833704    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb26f\" (UniqueName: \"kubernetes.io/projected/4498189b-b2cb-46e0-b20f-926be273f49a-kube-api-access-zb26f\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:39.833758 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.833725    5078 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-trusted-ca-bundle\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:39.833758 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.833736    5078 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4498189b-b2cb-46e0-b20f-926be273f49a-service-ca-bundle\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:39.833758 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:39.833744    5078 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4498189b-b2cb-46e0-b20f-926be273f49a-archives-path\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:40.564832 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:40.564759    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2"
Oct 30 23:54:40.565257 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:40.564759    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnrwh-lb8r2" event={"ID":"4498189b-b2cb-46e0-b20f-926be273f49a","Type":"ContainerDied","Data":"5b5c76ff8667d1a4ed4316ae7b3cfb23a0896bb6b80ab6241b555b11d182fc94"}
Oct 30 23:54:40.565257 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:40.564889    5078 scope.go:122] "RemoveContainer" containerID="b2207b815352754bc0a3c1df853d799b6a2f40fe5577eff1e8e70d4f817d4408"
Oct 30 23:54:40.585168 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:40.585146    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rnrwh-lb8r2"]
Oct 30 23:54:40.590680 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:40.590662    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rnrwh-lb8r2"]
Oct 30 23:54:42.192707 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:42.192670    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4498189b-b2cb-46e0-b20f-926be273f49a" path="/var/lib/kubelet/pods/4498189b-b2cb-46e0-b20f-926be273f49a/volumes"
Oct 30 23:54:55.503022 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.502971    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk" podUID="a5f9a68d-9d3a-4032-9f05-821e5563894a" containerName="registry" containerID="cri-o://9f9dedd71ade7e81904888f04bbfc66b5c7ddc82b6e64f18ad4328ce0574c9eb" gracePeriod=30
Oct 30 23:54:55.625237 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.625210    5078 generic.go:356] "Generic (PLEG): container finished" podID="a5f9a68d-9d3a-4032-9f05-821e5563894a" containerID="9f9dedd71ade7e81904888f04bbfc66b5c7ddc82b6e64f18ad4328ce0574c9eb" exitCode=0
Oct 30 23:54:55.625343 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.625278    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk" event={"ID":"a5f9a68d-9d3a-4032-9f05-821e5563894a","Type":"ContainerDied","Data":"9f9dedd71ade7e81904888f04bbfc66b5c7ddc82b6e64f18ad4328ce0574c9eb"}
Oct 30 23:54:55.802547 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.802521    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:55.959812 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.959788    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.959847    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.959868    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.959902    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.959930    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.959962    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960005 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.960000    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.960042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") pod \"a5f9a68d-9d3a-4032-9f05-821e5563894a\" (UID: \"a5f9a68d-9d3a-4032-9f05-821e5563894a\") "
Oct 30 23:54:55.960762 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.960699    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:55.960762 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.960722    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:55.962773 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.962746    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "kube-api-access-ng52f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:55.962892 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.962835    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:55.963027 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.963005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:55.963086 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.963014    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:55.963086 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.963044    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:55.967079 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:55.967052    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted" pod "a5f9a68d-9d3a-4032-9f05-821e5563894a" (UID: "a5f9a68d-9d3a-4032-9f05-821e5563894a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:56.061632 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061561    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng52f\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-kube-api-access-ng52f\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061632 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061591    5078 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-bound-sa-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061632 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061606    5078 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061632 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061617    5078 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-trusted-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061632 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061631    5078 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5f9a68d-9d3a-4032-9f05-821e5563894a-ca-trust-extracted\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061869 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061644    5078 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5f9a68d-9d3a-4032-9f05-821e5563894a-registry-certificates\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061869 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061657    5078 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-image-registry-private-configuration\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.061869 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.061672    5078 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5f9a68d-9d3a-4032-9f05-821e5563894a-installation-pull-secrets\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:56.631055 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.631015    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk" event={"ID":"a5f9a68d-9d3a-4032-9f05-821e5563894a","Type":"ContainerDied","Data":"81a66359229db7b1b5aa1c13ee057278e42320b6d39b6ca1ddbd9e09905a5a9e"}
Oct 30 23:54:56.631531 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.631081    5078 scope.go:122] "RemoveContainer" containerID="9f9dedd71ade7e81904888f04bbfc66b5c7ddc82b6e64f18ad4328ce0574c9eb"
Oct 30 23:54:56.631531 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.631090    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449bcf677-rqxsk"
Oct 30 23:54:56.658820 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.658793    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6449bcf677-rqxsk"]
Oct 30 23:54:56.662385 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:56.662363    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6449bcf677-rqxsk"]
Oct 30 23:54:58.195456 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:54:58.195421    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f9a68d-9d3a-4032-9f05-821e5563894a" path="/var/lib/kubelet/pods/a5f9a68d-9d3a-4032-9f05-821e5563894a/volumes"
Oct 30 23:55:06.384188 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:55:06.384145    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:55:17.720272 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:17.720240    5078 generic.go:356] "Generic (PLEG): container finished" podID="daf52a97-c6ca-4fba-a77b-67841614c162" containerID="24de10e9b98066fd7c9cc90b0bc230416b46473adbfe9f8dfa46dd2bf2155833" exitCode=0
Oct 30 23:55:17.720694 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:17.720318    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls" event={"ID":"daf52a97-c6ca-4fba-a77b-67841614c162","Type":"ContainerDied","Data":"24de10e9b98066fd7c9cc90b0bc230416b46473adbfe9f8dfa46dd2bf2155833"}
Oct 30 23:55:17.720694 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:17.720626    5078 scope.go:122] "RemoveContainer" containerID="24de10e9b98066fd7c9cc90b0bc230416b46473adbfe9f8dfa46dd2bf2155833"
Oct 30 23:55:18.726272 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:18.726237    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqgls" event={"ID":"daf52a97-c6ca-4fba-a77b-67841614c162","Type":"ContainerStarted","Data":"424e87c5e9243855900ca63471454408867755443803809367ab29ca385017c4"}
Oct 30 23:55:34.788689 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:34.788647    5078 generic.go:356] "Generic (PLEG): container finished" podID="967c5ac1-949a-4f15-94d6-a4fa8342ee73" containerID="7c66c2af5c59dd88127447f1546e95ae5de21070092a1478279ec9b15248d61d" exitCode=0
Oct 30 23:55:34.789054 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:34.788713    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gt7cj-g4r94" event={"ID":"967c5ac1-949a-4f15-94d6-a4fa8342ee73","Type":"ContainerDied","Data":"7c66c2af5c59dd88127447f1546e95ae5de21070092a1478279ec9b15248d61d"}
Oct 30 23:55:35.929303 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.929273    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:55:35.974234 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974206    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw\" (UniqueName: \"kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw\") pod \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") "
Oct 30 23:55:35.974362 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974290    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle\") pod \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") "
Oct 30 23:55:35.974362 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974314    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle\") pod \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") "
Oct 30 23:55:35.974362 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974337    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path\") pod \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\" (UID: \"967c5ac1-949a-4f15-94d6-a4fa8342ee73\") "
Oct 30 23:55:35.974735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974705    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path" pod "967c5ac1-949a-4f15-94d6-a4fa8342ee73" (UID: "967c5ac1-949a-4f15-94d6-a4fa8342ee73"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:55:35.974735 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974720    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle" pod "967c5ac1-949a-4f15-94d6-a4fa8342ee73" (UID: "967c5ac1-949a-4f15-94d6-a4fa8342ee73"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:35.975008 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.974740    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle" pod "967c5ac1-949a-4f15-94d6-a4fa8342ee73" (UID: "967c5ac1-949a-4f15-94d6-a4fa8342ee73"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:35.976362 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:35.976343    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw" pod "967c5ac1-949a-4f15-94d6-a4fa8342ee73" (UID: "967c5ac1-949a-4f15-94d6-a4fa8342ee73"). InnerVolumeSpecName "kube-api-access-lp7xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:36.074915 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.074827    5078 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/967c5ac1-949a-4f15-94d6-a4fa8342ee73-archives-path\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:36.074915 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.074860    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp7xw\" (UniqueName: \"kubernetes.io/projected/967c5ac1-949a-4f15-94d6-a4fa8342ee73-kube-api-access-lp7xw\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:36.074915 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.074871    5078 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-service-ca-bundle\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:36.074915 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.074880    5078 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967c5ac1-949a-4f15-94d6-a4fa8342ee73-trusted-ca-bundle\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:36.384139 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:55:36.384058    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:55:36.797895 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.797870    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gt7cj-g4r94"
Oct 30 23:55:36.798083 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.797910    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gt7cj-g4r94" event={"ID":"967c5ac1-949a-4f15-94d6-a4fa8342ee73","Type":"ContainerDied","Data":"163602b167787b3f61ed6f274db6e2aa5e5206024300b68f7d2f5d0f40abc288"}
Oct 30 23:55:36.798083 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:55:36.797952    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="163602b167787b3f61ed6f274db6e2aa5e5206024300b68f7d2f5d0f40abc288"
Oct 30 23:56:06.385430 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:56:06.385340    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:56:36.385145 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:56:36.385100    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:57:06.384887 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:57:06.384842    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:57:32.138604 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138526    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"]
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138913    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a5f9a68d-9d3a-4032-9f05-821e5563894a" containerName="registry"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138927    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f9a68d-9d3a-4032-9f05-821e5563894a" containerName="registry"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138939    5078 cpu_manager.go:414] "containerMap: removing container" podUID="967c5ac1-949a-4f15-94d6-a4fa8342ee73" containerName="insights-gathering"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138944    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c5ac1-949a-4f15-94d6-a4fa8342ee73" containerName="insights-gathering"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138955    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4498189b-b2cb-46e0-b20f-926be273f49a" containerName="insights-gathering"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138960    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4498189b-b2cb-46e0-b20f-926be273f49a" containerName="insights-gathering"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138972    5078 cpu_manager.go:414] "containerMap: removing container" podUID="36ab1743-a264-4439-a1a1-fd147ebd1248" containerName="registry"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.138976    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ab1743-a264-4439-a1a1-fd147ebd1248" containerName="registry"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.139030    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f9a68d-9d3a-4032-9f05-821e5563894a" containerName="registry"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.139038    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="967c5ac1-949a-4f15-94d6-a4fa8342ee73" containerName="insights-gathering"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.139045    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4498189b-b2cb-46e0-b20f-926be273f49a" containerName="insights-gathering"
Oct 30 23:57:32.139187 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.139051    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ab1743-a264-4439-a1a1-fd147ebd1248" containerName="registry"
Oct 30 23:57:32.141866 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.141845    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.159137 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.159119    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:57:32.159247 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.159200    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rlgpt\""
Oct 30 23:57:32.163129 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.163104    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"]
Oct 30 23:57:32.337045 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337010    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gs7k\" (UniqueName: \"kubernetes.io/projected/922f74f1-f495-4764-b2c9-1f40b2bcf849-kube-api-access-7gs7k\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337241 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337064    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337241 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337165    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337241 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337191    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337429 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337289    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337429 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337325    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337429 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337351    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337429 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337373    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.337643 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.337447    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f74f1-f495-4764-b2c9-1f40b2bcf849-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438370 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438298    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438370 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438330    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438370 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438346    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438645 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438497    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438645 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438528    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f74f1-f495-4764-b2c9-1f40b2bcf849-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438645 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438584    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gs7k\" (UniqueName: \"kubernetes.io/projected/922f74f1-f495-4764-b2c9-1f40b2bcf849-kube-api-access-7gs7k\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438810 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438704    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438810 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438733    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438810 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438791    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.438965 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.438855    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.439173 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.439144    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.439247 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.439233    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f74f1-f495-4764-b2c9-1f40b2bcf849-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.439283 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.439262    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.439319 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.439278    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.440717 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.440693    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.441064 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.441048    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.451724 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.451699    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gs7k\" (UniqueName: \"kubernetes.io/projected/922f74f1-f495-4764-b2c9-1f40b2bcf849-kube-api-access-7gs7k\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.451814 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.451770    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f74f1-f495-4764-b2c9-1f40b2bcf849-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b\" (UID: \"922f74f1-f495-4764-b2c9-1f40b2bcf849\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.455633 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.455617    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:32.731647 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:32.731627    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"]
Oct 30 23:57:32.733005 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:57:32.732983    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922f74f1_f495_4764_b2c9_1f40b2bcf849.slice/crio-197d0f1e62e804b851e038c56080b3b37475dd5d3584679aa473f5ebb074b62b WatchSource:0}: Error finding container 197d0f1e62e804b851e038c56080b3b37475dd5d3584679aa473f5ebb074b62b: Status 404 returned error can't find the container with id 197d0f1e62e804b851e038c56080b3b37475dd5d3584679aa473f5ebb074b62b
Oct 30 23:57:33.251384 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:33.251352    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b" event={"ID":"922f74f1-f495-4764-b2c9-1f40b2bcf849","Type":"ContainerStarted","Data":"197d0f1e62e804b851e038c56080b3b37475dd5d3584679aa473f5ebb074b62b"}
Oct 30 23:57:36.188331 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:36.188301    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:57:36.188665 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:36.188363    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:57:36.188665 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:36.188396    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:57:36.271756 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:36.271714    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b" event={"ID":"922f74f1-f495-4764-b2c9-1f40b2bcf849","Type":"ContainerStarted","Data":"ca5b634e9573f2516bc9ed0c95ad96e67ac613e8fd2837338e7bc669addc2500"}
Oct 30 23:57:36.301597 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:36.301553    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b" podStartSLOduration=0.848770167 podStartE2EDuration="4.301540295s" totalImagesPullingTime="3.452770128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:36.298936083 +0000 UTC m=+224.245306801" watchObservedRunningTime="2026-10-30 23:57:36.301540295 +0000 UTC m=+224.247911011"
Oct 30 23:57:36.387974 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:57:36.387939    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:57:36.456279 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:36.456213    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:37.461069 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:37.461040    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:38.280157 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:38.280128    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:57:38.280958 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:57:38.280939    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b"
Oct 30 23:58:00.701529 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.701499    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"]
Oct 30 23:58:00.706271 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.706248    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.715975 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.715946    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 23:58:00.716072 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.716027    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 23:58:00.732829 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.732797    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"]
Oct 30 23:58:00.745787 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.745767    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx98w\" (UniqueName: \"kubernetes.io/projected/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-kube-api-access-lx98w\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.745910 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.745822    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.745910 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.745847    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.746023 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.745938    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.746023 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.745959    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.746135 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.746040    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.777260 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.777236    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 23:58:00.777350 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.777335    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 23:58:00.777527 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.777510    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 23:58:00.777600 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.777565    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-475jf\""
Oct 30 23:58:00.847214 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847187    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.847313 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847228    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx98w\" (UniqueName: \"kubernetes.io/projected/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-kube-api-access-lx98w\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.847313 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847273    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.847313 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847293    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.847435 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847314    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.847435 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847356    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.847862 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.847838    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.849992 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.849971    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.850132 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.850114    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.850176 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.850136    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.850176 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.850113    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:00.860767 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:00.860749    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx98w\" (UniqueName: \"kubernetes.io/projected/654d9cdc-9ff9-4d6d-96e7-35fe4f90894f-kube-api-access-lx98w\") pod \"opendatahub-operator-controller-manager-58f644b8bd-tm95b\" (UID: \"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:01.021552 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:01.021482    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:01.323248 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:01.323224    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"]
Oct 30 23:58:01.324749 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:58:01.324720    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654d9cdc_9ff9_4d6d_96e7_35fe4f90894f.slice/crio-85158bb0b8fb6868e90c5e57365091afabe36701512b209f44a5bc90e985b1fb WatchSource:0}: Error finding container 85158bb0b8fb6868e90c5e57365091afabe36701512b209f44a5bc90e985b1fb: Status 404 returned error can't find the container with id 85158bb0b8fb6868e90c5e57365091afabe36701512b209f44a5bc90e985b1fb
Oct 30 23:58:01.368342 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:01.368315    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" event={"ID":"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f","Type":"ContainerStarted","Data":"85158bb0b8fb6868e90c5e57365091afabe36701512b209f44a5bc90e985b1fb"}
Oct 30 23:58:06.385457 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:58:06.385418    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:58:06.392437 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:06.392411    5078 generic.go:356] "Generic (PLEG): container finished" podID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerID="e753226c2824ec77342d586ccbcf4661c377392ca225b8ee9bfa55a452df0cc3" exitCode=0
Oct 30 23:58:06.392585 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:06.392534    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" event={"ID":"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f","Type":"ContainerDied","Data":"e753226c2824ec77342d586ccbcf4661c377392ca225b8ee9bfa55a452df0cc3"}
Oct 30 23:58:07.398699 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:07.398661    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" event={"ID":"654d9cdc-9ff9-4d6d-96e7-35fe4f90894f","Type":"ContainerStarted","Data":"37cb8118cf7b93298048dbf5a3e40d048a4f365b781640462225def9f6ca7444"}
Oct 30 23:58:07.399131 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:07.398804    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:07.458437 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:07.458389    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podStartSLOduration=2.540323186 podStartE2EDuration="7.45837608s" totalImagesPullingTime="4.501054132s" totalInitContainerRuntime="416.998762ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:07.45611483 +0000 UTC m=+255.402485547" watchObservedRunningTime="2026-10-30 23:58:07.45837608 +0000 UTC m=+255.404746790"
Oct 30 23:58:16.567937 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.567900    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"]
Oct 30 23:58:16.574698 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.574672    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:16.583416 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.583396    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"]
Oct 30 23:58:16.583814 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.583791    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 23:58:16.583916 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.583902    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 23:58:16.584056 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.584041    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j8twh\""
Oct 30 23:58:16.677391 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.677365    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7lx\" (UniqueName: \"kubernetes.io/projected/bf1eaf8e-40da-42fe-bb27-6900f2488745-kube-api-access-4v7lx\") pod \"authorino-operator-7b7bb4bc77-ljnt9\" (UID: \"bf1eaf8e-40da-42fe-bb27-6900f2488745\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:16.778237 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.778203    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7lx\" (UniqueName: \"kubernetes.io/projected/bf1eaf8e-40da-42fe-bb27-6900f2488745-kube-api-access-4v7lx\") pod \"authorino-operator-7b7bb4bc77-ljnt9\" (UID: \"bf1eaf8e-40da-42fe-bb27-6900f2488745\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:16.793213 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.793186    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7lx\" (UniqueName: \"kubernetes.io/projected/bf1eaf8e-40da-42fe-bb27-6900f2488745-kube-api-access-4v7lx\") pod \"authorino-operator-7b7bb4bc77-ljnt9\" (UID: \"bf1eaf8e-40da-42fe-bb27-6900f2488745\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:16.892419 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:16.892398    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:17.200130 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:17.200061    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"]
Oct 30 23:58:17.203639 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:58:17.203613    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1eaf8e_40da_42fe_bb27_6900f2488745.slice/crio-b3453e4332eea6de086b55ce9a6bc5d634af1e60dd025913d341c8131b62a7ef WatchSource:0}: Error finding container b3453e4332eea6de086b55ce9a6bc5d634af1e60dd025913d341c8131b62a7ef: Status 404 returned error can't find the container with id b3453e4332eea6de086b55ce9a6bc5d634af1e60dd025913d341c8131b62a7ef
Oct 30 23:58:17.438602 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:17.438569    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" event={"ID":"bf1eaf8e-40da-42fe-bb27-6900f2488745","Type":"ContainerStarted","Data":"b3453e4332eea6de086b55ce9a6bc5d634af1e60dd025913d341c8131b62a7ef"}
Oct 30 23:58:18.405402 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:18.405374    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 30 23:58:21.457861 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:21.457825    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" event={"ID":"bf1eaf8e-40da-42fe-bb27-6900f2488745","Type":"ContainerStarted","Data":"e7e6133ee19f34df4a5dd9f4849410333a6e88347215a342cacd944d8a0ea28d"}
Oct 30 23:58:21.458299 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:21.457929    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:21.485859 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:21.485812    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podStartSLOduration=2.118653056 podStartE2EDuration="5.4857998s" totalImagesPullingTime="3.367146744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:21.484345442 +0000 UTC m=+269.430716160" watchObservedRunningTime="2026-10-30 23:58:21.4857998 +0000 UTC m=+269.432170517"
Oct 30 23:58:28.333896 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.333858    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"]
Oct 30 23:58:28.337580 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.337559    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.344231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.344211    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-frf6h\""
Oct 30 23:58:28.352776 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.352753    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"]
Oct 30 23:58:28.476520 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.476495    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhwz\" (UniqueName: \"kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz\") pod \"kuadrant-operator-controller-manager-678d87766f-wmwmw\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.476681 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.476528    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wmwmw\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.576899 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.576870    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhwz\" (UniqueName: \"kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz\") pod \"kuadrant-operator-controller-manager-678d87766f-wmwmw\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.577049 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.576909    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wmwmw\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.577340 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.577322    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wmwmw\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.590226 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.590162    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhwz\" (UniqueName: \"kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz\") pod \"kuadrant-operator-controller-manager-678d87766f-wmwmw\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.650163 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.650137    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:28.929827 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:28.929804    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"]
Oct 30 23:58:28.931831 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:58:28.931806    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873cbdbb_6751_42e3_8c82_c997ed53e762.slice/crio-003a6ff3a57d40f6462592e73cf95324306e19d4f7bf248523ea5253dce03b37 WatchSource:0}: Error finding container 003a6ff3a57d40f6462592e73cf95324306e19d4f7bf248523ea5253dce03b37: Status 404 returned error can't find the container with id 003a6ff3a57d40f6462592e73cf95324306e19d4f7bf248523ea5253dce03b37
Oct 30 23:58:29.493116 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:29.493081    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw" event={"ID":"873cbdbb-6751-42e3-8c82-c997ed53e762","Type":"ContainerStarted","Data":"003a6ff3a57d40f6462592e73cf95324306e19d4f7bf248523ea5253dce03b37"}
Oct 30 23:58:32.465023 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:32.464990    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 30 23:58:33.014001 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.013966    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"]
Oct 30 23:58:33.018098 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.017756    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.033289 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.033246    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zrc8x\""
Oct 30 23:58:33.034664 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.034642    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"]
Oct 30 23:58:33.113067 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113297 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113105    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113297 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113132    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113297 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113156    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113297 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113207    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c108abe8-dc34-426e-9a4e-becaf4e882b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113297 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113245    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpjr\" (UniqueName: \"kubernetes.io/projected/c108abe8-dc34-426e-9a4e-becaf4e882b5-kube-api-access-ggpjr\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113297 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113274    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113566 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.113566 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.113347    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.214727 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214701    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c108abe8-dc34-426e-9a4e-becaf4e882b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.214873 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214740    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpjr\" (UniqueName: \"kubernetes.io/projected/c108abe8-dc34-426e-9a4e-becaf4e882b5-kube-api-access-ggpjr\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.214873 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214766    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.214873 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214811    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.214873 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214835    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215071 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214881    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215071 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214923    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215071 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215071 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.214967    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215256 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.215238    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215452 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.215414    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215596 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.215445    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215596 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.215507    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.215596 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.215558    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c108abe8-dc34-426e-9a4e-becaf4e882b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.217446 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.217422    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.217609 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.217591    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.228673 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.228645    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpjr\" (UniqueName: \"kubernetes.io/projected/c108abe8-dc34-426e-9a4e-becaf4e882b5-kube-api-access-ggpjr\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.229368 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.229349    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c108abe8-dc34-426e-9a4e-becaf4e882b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv\" (UID: \"c108abe8-dc34-426e-9a4e-becaf4e882b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.342159 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.342068    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:33.619342 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.619320    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"]
Oct 30 23:58:33.620644 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:58:33.620622    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc108abe8_dc34_426e_9a4e_becaf4e882b5.slice/crio-c4b1d021ed8770243ae2eb19cf4c25bcc81469489cd152cdee11e922433b202b WatchSource:0}: Error finding container c4b1d021ed8770243ae2eb19cf4c25bcc81469489cd152cdee11e922433b202b: Status 404 returned error can't find the container with id c4b1d021ed8770243ae2eb19cf4c25bcc81469489cd152cdee11e922433b202b
Oct 30 23:58:33.622668 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.622637    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:33.622753 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.622699    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:33.622753 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:33.622726    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:34.514829 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.514783    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" event={"ID":"c108abe8-dc34-426e-9a4e-becaf4e882b5","Type":"ContainerStarted","Data":"c1d50b6e67339eea37b45642a5c60d44f59b13b04077d05838eac84cf45ae24f"}
Oct 30 23:58:34.515009 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.514860    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" event={"ID":"c108abe8-dc34-426e-9a4e-becaf4e882b5","Type":"ContainerStarted","Data":"c4b1d021ed8770243ae2eb19cf4c25bcc81469489cd152cdee11e922433b202b"}
Oct 30 23:58:34.551011 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.550965    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" podStartSLOduration=2.550951617 podStartE2EDuration="2.550951617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:34.550448029 +0000 UTC m=+282.496818746" watchObservedRunningTime="2026-10-30 23:58:34.550951617 +0000 UTC m=+282.497322333"
Oct 30 23:58:34.894199 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.894169    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"]
Oct 30 23:58:34.900529 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.900509    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:34.909550 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.909455    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:58:34.909550 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.909479    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:58:34.909550 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.909455    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4zs72\""
Oct 30 23:58:34.914551 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.914528    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 23:58:34.916487 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.916449    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"]
Oct 30 23:58:34.931835 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.931812    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9bb7055-cf51-4182-a092-2364ed3d5444-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:34.931951 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.931872    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9bb7055-cf51-4182-a092-2364ed3d5444-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:34.931951 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.931902    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9bb7055-cf51-4182-a092-2364ed3d5444-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:34.932059 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:34.931970    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49zt\" (UniqueName: \"kubernetes.io/projected/e9bb7055-cf51-4182-a092-2364ed3d5444-kube-api-access-w49zt\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.032766 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.032734    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9bb7055-cf51-4182-a092-2364ed3d5444-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.032962 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.032778    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9bb7055-cf51-4182-a092-2364ed3d5444-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.032962 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.032798    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49zt\" (UniqueName: \"kubernetes.io/projected/e9bb7055-cf51-4182-a092-2364ed3d5444-kube-api-access-w49zt\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.032962 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.032849    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9bb7055-cf51-4182-a092-2364ed3d5444-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.033238 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.033219    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9bb7055-cf51-4182-a092-2364ed3d5444-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.033306 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.033242    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9bb7055-cf51-4182-a092-2364ed3d5444-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.033667 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.033649    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9bb7055-cf51-4182-a092-2364ed3d5444-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.063220 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.063187    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49zt\" (UniqueName: \"kubernetes.io/projected/e9bb7055-cf51-4182-a092-2364ed3d5444-kube-api-access-w49zt\") pod \"kserve-module-controller-manager-7566cf5fc8-mpl9v\" (UID: \"e9bb7055-cf51-4182-a092-2364ed3d5444\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.218225 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.218133    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:35.342743 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.342713    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:35.347223 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.347203    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:35.519997 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.519930    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:35.520823 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.520805    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv"
Oct 30 23:58:35.534496 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:35.534477    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"]
Oct 30 23:58:35.536368 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:58:35.536347    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bb7055_cf51_4182_a092_2364ed3d5444.slice/crio-1ab7d4c6fe69dbc1c782cb4d8d3c3ed10a6d94f506e7c92121f94d9eaeac5c72 WatchSource:0}: Error finding container 1ab7d4c6fe69dbc1c782cb4d8d3c3ed10a6d94f506e7c92121f94d9eaeac5c72: Status 404 returned error can't find the container with id 1ab7d4c6fe69dbc1c782cb4d8d3c3ed10a6d94f506e7c92121f94d9eaeac5c72
Oct 30 23:58:36.395366 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:58:36.395325    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:58:36.529361 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:36.529326    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" event={"ID":"e9bb7055-cf51-4182-a092-2364ed3d5444","Type":"ContainerStarted","Data":"1ab7d4c6fe69dbc1c782cb4d8d3c3ed10a6d94f506e7c92121f94d9eaeac5c72"}
Oct 30 23:58:38.541557 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:38.541513    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" event={"ID":"e9bb7055-cf51-4182-a092-2364ed3d5444","Type":"ContainerStarted","Data":"dbd38a7138471dca14bc467e687c3f54e74c3572af13eaf29768ff1418d1eb12"}
Oct 30 23:58:38.542080 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:38.541589    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:58:38.570411 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:38.570352    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podStartSLOduration=2.009216282 podStartE2EDuration="4.570332715s" totalImagesPullingTime="2.561116433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:38.567092359 +0000 UTC m=+286.513463077" watchObservedRunningTime="2026-10-30 23:58:38.570332715 +0000 UTC m=+286.516703433"
Oct 30 23:58:39.547148 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:39.547109    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw" event={"ID":"873cbdbb-6751-42e3-8c82-c997ed53e762","Type":"ContainerStarted","Data":"32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4"}
Oct 30 23:58:39.547602 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:39.547361    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:39.575269 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:39.575223    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw" podStartSLOduration=1.8380643239999999 podStartE2EDuration="11.575213609s" totalImagesPullingTime="9.737149285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:39.57499247 +0000 UTC m=+287.521363186" watchObservedRunningTime="2026-10-30 23:58:39.575213609 +0000 UTC m=+287.521584325"
Oct 30 23:58:50.554369 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:50.554338    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 30 23:58:52.208914 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:52.208883    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:58:52.215335 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:52.215313    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 30 23:58:59.548601 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:58:59.548570    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 30 23:59:06.384255 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:59:06.384179    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:59:20.774955 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.774914    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"]
Oct 30 23:59:20.782531 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.782505    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:20.800449 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.800422    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"]
Oct 30 23:59:20.801292 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.801271    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 23:59:20.801426 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.801357    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 23:59:20.801690 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.801672    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 23:59:20.801763 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.801724    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:59:20.801819 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.801812    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 23:59:20.801947 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.801933    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-l65ww\""
Oct 30 23:59:20.917042 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.917012    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-cert\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:20.917164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.917052    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-manager-config\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:20.917164 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.917146    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvjd\" (UniqueName: \"kubernetes.io/projected/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-kube-api-access-8bvjd\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:20.917240 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:20.917194    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-metrics-certs\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.023813 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.023766    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-cert\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.024555 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.024530    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-manager-config\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.024686 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.024602    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvjd\" (UniqueName: \"kubernetes.io/projected/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-kube-api-access-8bvjd\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.024686 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.024656    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-metrics-certs\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.026064 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.025968    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-manager-config\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.030439 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.030414    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-cert\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.031934 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.031908    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-metrics-certs\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.041231 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.041208    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvjd\" (UniqueName: \"kubernetes.io/projected/3c0d9ceb-7c2b-4872-a065-4bfe4b332064-kube-api-access-8bvjd\") pod \"jobset-controller-manager-657fcc8b8f-z9psz\" (UID: \"3c0d9ceb-7c2b-4872-a065-4bfe4b332064\") " pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.093833 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.093813    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:21.370974 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.370954    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"]
Oct 30 23:59:21.373162 ip-10-0-143-16 kubenswrapper[5078]: W1030 23:59:21.373133    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0d9ceb_7c2b_4872_a065_4bfe4b332064.slice/crio-b4f26159348ddf02a332864f9d4754c6276d4a314fa7cc2a82b554afdbe8be32 WatchSource:0}: Error finding container b4f26159348ddf02a332864f9d4754c6276d4a314fa7cc2a82b554afdbe8be32: Status 404 returned error can't find the container with id b4f26159348ddf02a332864f9d4754c6276d4a314fa7cc2a82b554afdbe8be32
Oct 30 23:59:21.375018 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.374997    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:59:21.728493 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:21.728443    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" event={"ID":"3c0d9ceb-7c2b-4872-a065-4bfe4b332064","Type":"ContainerStarted","Data":"b4f26159348ddf02a332864f9d4754c6276d4a314fa7cc2a82b554afdbe8be32"}
Oct 30 23:59:25.746870 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:25.746834    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" event={"ID":"3c0d9ceb-7c2b-4872-a065-4bfe4b332064","Type":"ContainerStarted","Data":"89597f02c6c654dd72b20e98d8d535d4d2311ad7d44e1b6083f0a6fead8c7eb2"}
Oct 30 23:59:25.747270 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:25.746961    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 30 23:59:25.770759 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:25.770713    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podStartSLOduration=2.374504004 podStartE2EDuration="5.770699104s" totalImagesPullingTime="3.3961951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:25.768642484 +0000 UTC m=+333.715013203" watchObservedRunningTime="2026-10-30 23:59:25.770699104 +0000 UTC m=+333.717069822"
Oct 30 23:59:36.384888 ip-10-0-143-16 kubenswrapper[5078]: E1030 23:59:36.384850    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 30 23:59:36.757415 ip-10-0-143-16 kubenswrapper[5078]: I1030 23:59:36.757335    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 31 00:00:06.385565 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:00:06.385529    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:00:36.385389 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:00:36.385299    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:01:06.384994 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:01:06.384959    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:01:36.384189 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:01:36.384152    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:02:06.385196 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:02:06.385112    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:02:36.384883 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:02:36.384848    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:03:06.386100 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:03:06.386056    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:03:36.384994 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:03:36.384915    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:03:52.235354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:03:52.235327    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:03:52.242129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:03:52.242108    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:04:06.384636 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:04:06.384595    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:04:36.384763 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:04:36.384727    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:05:06.385072 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:05:06.384990    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:05:36.384848 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:05:36.384809    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:06:06.385033 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:06:06.384994    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:06:36.385377 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:06:36.385295    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:07:06.385316 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:07:06.385271    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:07:36.385423 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:07:36.385386    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:08:06.384393 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:08:06.384316    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:08:36.385952 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:08:36.385845    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:08:52.261362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:52.261335    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:08:52.269637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:52.269614    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:08:53.624113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.624085    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wmwmw_873cbdbb-6751-42e3-8c82-c997ed53e762/manager/0.log"
Oct 31 00:08:53.820770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.820738    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"]
Oct 31 00:08:53.821012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.820969    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw" podUID="873cbdbb-6751-42e3-8c82-c997ed53e762" containerName="manager" containerID="cri-o://32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4" gracePeriod=10
Oct 31 00:08:53.853670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.853646    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"]
Oct 31 00:08:53.857062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.857046    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:53.871643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.871617    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"]
Oct 31 00:08:53.955031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.954991    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvgk\" (UniqueName: \"kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk\") pod \"kuadrant-operator-controller-manager-678d87766f-l5xz9\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:53.955144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:53.955113    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l5xz9\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:54.056152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.056121    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvgk\" (UniqueName: \"kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk\") pod \"kuadrant-operator-controller-manager-678d87766f-l5xz9\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:54.056313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.056190    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l5xz9\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:54.056830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.056803    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l5xz9\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:54.069445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.069424    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvgk\" (UniqueName: \"kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk\") pod \"kuadrant-operator-controller-manager-678d87766f-l5xz9\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:54.085133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.085111    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 31 00:08:54.157046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.156959    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume\") pod \"873cbdbb-6751-42e3-8c82-c997ed53e762\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") "
Oct 31 00:08:54.157046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.156997    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz\" (UniqueName: \"kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz\") pod \"873cbdbb-6751-42e3-8c82-c997ed53e762\" (UID: \"873cbdbb-6751-42e3-8c82-c997ed53e762\") "
Oct 31 00:08:54.157325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.157300    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume" pod "873cbdbb-6751-42e3-8c82-c997ed53e762" (UID: "873cbdbb-6751-42e3-8c82-c997ed53e762"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:54.159377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.159353    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz" pod "873cbdbb-6751-42e3-8c82-c997ed53e762" (UID: "873cbdbb-6751-42e3-8c82-c997ed53e762"). InnerVolumeSpecName "kube-api-access-5zhwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:54.194071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.194044    5078 generic.go:356] "Generic (PLEG): container finished" podID="873cbdbb-6751-42e3-8c82-c997ed53e762" containerID="32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4" exitCode=0
Oct 31 00:08:54.194211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.194097    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"
Oct 31 00:08:54.194211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.194136    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw" event={"ID":"873cbdbb-6751-42e3-8c82-c997ed53e762","Type":"ContainerDied","Data":"32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4"}
Oct 31 00:08:54.194211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.194187    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw" event={"ID":"873cbdbb-6751-42e3-8c82-c997ed53e762","Type":"ContainerDied","Data":"003a6ff3a57d40f6462592e73cf95324306e19d4f7bf248523ea5253dce03b37"}
Oct 31 00:08:54.194211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.194209    5078 scope.go:122] "RemoveContainer" containerID="32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4"
Oct 31 00:08:54.203871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.203854    5078 scope.go:122] "RemoveContainer" containerID="32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4"
Oct 31 00:08:54.204149 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:08:54.204130    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4\": container with ID starting with 32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4 not found: ID does not exist" containerID="32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4"
Oct 31 00:08:54.204222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.204161    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4"} err="failed to get container status \"32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4\": rpc error: code = NotFound desc = could not find container \"32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4\": container with ID starting with 32fe75bba245ec5571507af1432e1ce11aa3752a41eef487fc39257231875bc4 not found: ID does not exist"
Oct 31 00:08:54.210322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.210303    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:54.227510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.227489    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"]
Oct 31 00:08:54.236394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.236372    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wmwmw"]
Oct 31 00:08:54.257917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.257892    5078 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873cbdbb-6751-42e3-8c82-c997ed53e762-extensions-socket-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.257917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.257917    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhwz\" (UniqueName: \"kubernetes.io/projected/873cbdbb-6751-42e3-8c82-c997ed53e762-kube-api-access-5zhwz\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.501269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:54.501239    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"]
Oct 31 00:08:54.502936 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:08:54.502913    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e84018f_f7a2_4588_bd5b_b67814c509bd.slice/crio-d63e44546c3664762b8161c405dc36d3973989622271fa8218569681dbd6482b WatchSource:0}: Error finding container d63e44546c3664762b8161c405dc36d3973989622271fa8218569681dbd6482b: Status 404 returned error can't find the container with id d63e44546c3664762b8161c405dc36d3973989622271fa8218569681dbd6482b
Oct 31 00:08:55.200914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:55.200876    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" event={"ID":"7e84018f-f7a2-4588-bd5b-b67814c509bd","Type":"ContainerStarted","Data":"e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"}
Oct 31 00:08:55.201314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:55.200922    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" event={"ID":"7e84018f-f7a2-4588-bd5b-b67814c509bd","Type":"ContainerStarted","Data":"d63e44546c3664762b8161c405dc36d3973989622271fa8218569681dbd6482b"}
Oct 31 00:08:55.201314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:55.200954    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:08:55.231233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:55.231182    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podStartSLOduration=2.231167727 podStartE2EDuration="2.231167727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:55.228494697 +0000 UTC m=+903.174865414" watchObservedRunningTime="2026-10-31 00:08:55.231167727 +0000 UTC m=+903.177538444"
Oct 31 00:08:56.192656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:08:56.192620    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873cbdbb-6751-42e3-8c82-c997ed53e762" path="/var/lib/kubelet/pods/873cbdbb-6751-42e3-8c82-c997ed53e762/volumes"
Oct 31 00:09:06.208138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:06.208109    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:09:06.385187 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:09:06.385153    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:09:13.189736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.189702    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mccbb"]
Oct 31 00:09:13.190157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.190112    5078 cpu_manager.go:414] "containerMap: removing container" podUID="873cbdbb-6751-42e3-8c82-c997ed53e762" containerName="manager"
Oct 31 00:09:13.190157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.190126    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873cbdbb-6751-42e3-8c82-c997ed53e762" containerName="manager"
Oct 31 00:09:13.190240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.190228    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="873cbdbb-6751-42e3-8c82-c997ed53e762" containerName="manager"
Oct 31 00:09:13.193435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.193413    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:13.201403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.201249    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mccbb"]
Oct 31 00:09:13.203434 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.203413    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gzz8j\""
Oct 31 00:09:13.314089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.314058    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m55s\" (UniqueName: \"kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s\") pod \"authorino-57f7f8659f-mccbb\" (UID: \"9014187c-5388-414e-9ae4-041445f9d9ae\") " pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:13.415129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.415096    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m55s\" (UniqueName: \"kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s\") pod \"authorino-57f7f8659f-mccbb\" (UID: \"9014187c-5388-414e-9ae4-041445f9d9ae\") " pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:13.430987 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.430951    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m55s\" (UniqueName: \"kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s\") pod \"authorino-57f7f8659f-mccbb\" (UID: \"9014187c-5388-414e-9ae4-041445f9d9ae\") " pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:13.522228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.522154    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:13.796184 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.796158    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mccbb"]
Oct 31 00:09:13.798269 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:09:13.798243    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9014187c_5388_414e_9ae4_041445f9d9ae.slice/crio-9a9c2b681b3e7f9f3b46a26f4849980cf7d484bf88adaed7024dda1a448a5441 WatchSource:0}: Error finding container 9a9c2b681b3e7f9f3b46a26f4849980cf7d484bf88adaed7024dda1a448a5441: Status 404 returned error can't find the container with id 9a9c2b681b3e7f9f3b46a26f4849980cf7d484bf88adaed7024dda1a448a5441
Oct 31 00:09:13.799603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:13.799584    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:09:14.285714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:14.285675    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mccbb" event={"ID":"9014187c-5388-414e-9ae4-041445f9d9ae","Type":"ContainerStarted","Data":"9a9c2b681b3e7f9f3b46a26f4849980cf7d484bf88adaed7024dda1a448a5441"}
Oct 31 00:09:16.295966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:16.295933    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mccbb" event={"ID":"9014187c-5388-414e-9ae4-041445f9d9ae","Type":"ContainerStarted","Data":"01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d"}
Oct 31 00:09:16.314308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:16.314257    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-mccbb" podStartSLOduration=1.007963177 podStartE2EDuration="3.314243729s" totalImagesPullingTime="2.306280552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:16.314077876 +0000 UTC m=+924.260448594" watchObservedRunningTime="2026-10-31 00:09:16.314243729 +0000 UTC m=+924.260614446"
Oct 31 00:09:17.775635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:17.775603    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mccbb"]
Oct 31 00:09:18.304673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:18.304617    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-mccbb" podUID="9014187c-5388-414e-9ae4-041445f9d9ae" containerName="authorino" containerID="cri-o://01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d" gracePeriod=30
Oct 31 00:09:18.546778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:18.546749    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:18.660213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:18.660182    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s\" (UniqueName: \"kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s\") pod \"9014187c-5388-414e-9ae4-041445f9d9ae\" (UID: \"9014187c-5388-414e-9ae4-041445f9d9ae\") "
Oct 31 00:09:18.662275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:18.662254    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s" pod "9014187c-5388-414e-9ae4-041445f9d9ae" (UID: "9014187c-5388-414e-9ae4-041445f9d9ae"). InnerVolumeSpecName "kube-api-access-6m55s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:18.760952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:18.760924    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m55s\" (UniqueName: \"kubernetes.io/projected/9014187c-5388-414e-9ae4-041445f9d9ae-kube-api-access-6m55s\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:19.310437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.310397    5078 generic.go:356] "Generic (PLEG): container finished" podID="9014187c-5388-414e-9ae4-041445f9d9ae" containerID="01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d" exitCode=0
Oct 31 00:09:19.310894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.310442    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mccbb"
Oct 31 00:09:19.310894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.310454    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mccbb" event={"ID":"9014187c-5388-414e-9ae4-041445f9d9ae","Type":"ContainerDied","Data":"01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d"}
Oct 31 00:09:19.310894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.310506    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mccbb" event={"ID":"9014187c-5388-414e-9ae4-041445f9d9ae","Type":"ContainerDied","Data":"9a9c2b681b3e7f9f3b46a26f4849980cf7d484bf88adaed7024dda1a448a5441"}
Oct 31 00:09:19.310894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.310528    5078 scope.go:122] "RemoveContainer" containerID="01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d"
Oct 31 00:09:19.319723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.319703    5078 scope.go:122] "RemoveContainer" containerID="01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d"
Oct 31 00:09:19.319968 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:09:19.319949    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d\": container with ID starting with 01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d not found: ID does not exist" containerID="01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d"
Oct 31 00:09:19.320031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.319981    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d"} err="failed to get container status \"01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d\": rpc error: code = NotFound desc = could not find container \"01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d\": container with ID starting with 01ca74a4f31240bbd9b16efd27851d225cdc879d953f54dc28bfcc2e74f6972d not found: ID does not exist"
Oct 31 00:09:19.333018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.332995    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mccbb"]
Oct 31 00:09:19.339026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:19.339007    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mccbb"]
Oct 31 00:09:20.192617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:20.192584    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9014187c-5388-414e-9ae4-041445f9d9ae" path="/var/lib/kubelet/pods/9014187c-5388-414e-9ae4-041445f9d9ae/volumes"
Oct 31 00:09:36.385854 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:09:36.385774    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:09:48.801634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.801600    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdf75"]
Oct 31 00:09:48.802011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.801971    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9014187c-5388-414e-9ae4-041445f9d9ae" containerName="authorino"
Oct 31 00:09:48.802011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.801983    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9014187c-5388-414e-9ae4-041445f9d9ae" containerName="authorino"
Oct 31 00:09:48.802147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.802049    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9014187c-5388-414e-9ae4-041445f9d9ae" containerName="authorino"
Oct 31 00:09:48.805230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.805211    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:48.810910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.810885    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:09:48.812961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.812940    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k4pdx\""
Oct 31 00:09:48.812961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.812940    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 00:09:48.813167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.812995    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 00:09:48.816947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.816928    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdf75"]
Oct 31 00:09:48.897016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.896986    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgpw\" (UniqueName: \"kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:48.897155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.897039    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:48.897221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.897165    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:48.997874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.997840    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgpw\" (UniqueName: \"kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:48.998045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.997890    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:48.998045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:48.997930    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:49.000514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.000489    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:49.000739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.000717    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:49.012154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.012127    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgpw\" (UniqueName: \"kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw\") pod \"maas-controller-579cf7c6d-rdf75\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") " pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:49.110937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.110866    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-jrv29"]
Oct 31 00:09:49.114379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.114364    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.118872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.118858    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:49.133418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.133393    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-jrv29"]
Oct 31 00:09:49.200762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.200729    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.200941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.200789    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfphz\" (UniqueName: \"kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.201005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.200943    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.302284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.302254    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.302432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.302329    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.302432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.302370    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfphz\" (UniqueName: \"kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.305116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.305087    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.305226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.305113    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.315503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.315453    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfphz\" (UniqueName: \"kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz\") pod \"maas-controller-7b8bb6f8d-jrv29\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") " pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.398870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.398844    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdf75"]
Oct 31 00:09:49.400064 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:09:49.400040    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3ba51b_28cf_4496_a719_857d9e2e3763.slice/crio-5d29762003d8f66741fab8dba3a4af359bdd57ff85aa1515e28bf89545b90432 WatchSource:0}: Error finding container 5d29762003d8f66741fab8dba3a4af359bdd57ff85aa1515e28bf89545b90432: Status 404 returned error can't find the container with id 5d29762003d8f66741fab8dba3a4af359bdd57ff85aa1515e28bf89545b90432
Oct 31 00:09:49.429036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.429008    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:49.440423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.440399    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdf75" event={"ID":"7d3ba51b-28cf-4496-a719-857d9e2e3763","Type":"ContainerStarted","Data":"5d29762003d8f66741fab8dba3a4af359bdd57ff85aa1515e28bf89545b90432"}
Oct 31 00:09:49.706599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:49.706570    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-jrv29"]
Oct 31 00:09:49.709546 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:09:49.709516    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cb91e9_c6e4_460f_9318_49d79080dedb.slice/crio-b2c810beb59dffec8927ce2cf7315c7e1075c61918a13449253a876196bfacbe WatchSource:0}: Error finding container b2c810beb59dffec8927ce2cf7315c7e1075c61918a13449253a876196bfacbe: Status 404 returned error can't find the container with id b2c810beb59dffec8927ce2cf7315c7e1075c61918a13449253a876196bfacbe
Oct 31 00:09:50.456148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:50.456109    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" event={"ID":"68cb91e9-c6e4-460f-9318-49d79080dedb","Type":"ContainerStarted","Data":"b2c810beb59dffec8927ce2cf7315c7e1075c61918a13449253a876196bfacbe"}
Oct 31 00:09:52.382323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:52.382303    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:09:53.476900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:53.476869    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdf75" event={"ID":"7d3ba51b-28cf-4496-a719-857d9e2e3763","Type":"ContainerStarted","Data":"d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96"}
Oct 31 00:09:53.477360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:53.476932    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:09:53.478554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:53.478523    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" event={"ID":"68cb91e9-c6e4-460f-9318-49d79080dedb","Type":"ContainerStarted","Data":"27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"}
Oct 31 00:09:53.478721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:53.478581    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:09:53.502721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:53.502672    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-rdf75" podStartSLOduration=2.526577022 podStartE2EDuration="5.50265974s" totalImagesPullingTime="2.976082718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:53.50140824 +0000 UTC m=+961.447778973" watchObservedRunningTime="2026-10-31 00:09:53.50265974 +0000 UTC m=+961.449030457"
Oct 31 00:09:53.547616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:09:53.547574    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podStartSLOduration=1.7228354970000002 podStartE2EDuration="4.547560628s" totalImagesPullingTime="2.824725131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:53.542951513 +0000 UTC m=+961.489322232" watchObservedRunningTime="2026-10-31 00:09:53.547560628 +0000 UTC m=+961.493931352"
Oct 31 00:10:04.484884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:04.484849    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:10:04.485317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:04.484927    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:10:04.577792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:04.577746    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdf75"]
Oct 31 00:10:04.578026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:04.577972    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-rdf75" podUID="7d3ba51b-28cf-4496-a719-857d9e2e3763" containerName="manager" containerID="cri-o://d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96" gracePeriod=30
Oct 31 00:10:04.921322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:04.921299    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:10:05.035716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.035680    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert\") pod \"7d3ba51b-28cf-4496-a719-857d9e2e3763\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") "
Oct 31 00:10:05.035901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.035749    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs\") pod \"7d3ba51b-28cf-4496-a719-857d9e2e3763\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") "
Oct 31 00:10:05.035901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.035790    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw\" (UniqueName: \"kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw\") pod \"7d3ba51b-28cf-4496-a719-857d9e2e3763\" (UID: \"7d3ba51b-28cf-4496-a719-857d9e2e3763\") "
Oct 31 00:10:05.038183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.038150    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert" pod "7d3ba51b-28cf-4496-a719-857d9e2e3763" (UID: "7d3ba51b-28cf-4496-a719-857d9e2e3763"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:10:05.038313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.038202    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw" pod "7d3ba51b-28cf-4496-a719-857d9e2e3763" (UID: "7d3ba51b-28cf-4496-a719-857d9e2e3763"). InnerVolumeSpecName "kube-api-access-hbgpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:10:05.038313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.038210    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs" pod "7d3ba51b-28cf-4496-a719-857d9e2e3763" (UID: "7d3ba51b-28cf-4496-a719-857d9e2e3763"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:10:05.136683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.136658    5078 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-webhook-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:05.136840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.136684    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d3ba51b-28cf-4496-a719-857d9e2e3763-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:05.136840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.136696    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbgpw\" (UniqueName: \"kubernetes.io/projected/7d3ba51b-28cf-4496-a719-857d9e2e3763-kube-api-access-hbgpw\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:05.533347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.533266    5078 generic.go:356] "Generic (PLEG): container finished" podID="7d3ba51b-28cf-4496-a719-857d9e2e3763" containerID="d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96" exitCode=0
Oct 31 00:10:05.533347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.533328    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdf75"
Oct 31 00:10:05.533820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.533353    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdf75" event={"ID":"7d3ba51b-28cf-4496-a719-857d9e2e3763","Type":"ContainerDied","Data":"d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96"}
Oct 31 00:10:05.533820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.533438    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdf75" event={"ID":"7d3ba51b-28cf-4496-a719-857d9e2e3763","Type":"ContainerDied","Data":"5d29762003d8f66741fab8dba3a4af359bdd57ff85aa1515e28bf89545b90432"}
Oct 31 00:10:05.533820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.533483    5078 scope.go:122] "RemoveContainer" containerID="d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96"
Oct 31 00:10:05.543274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.543259    5078 scope.go:122] "RemoveContainer" containerID="d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96"
Oct 31 00:10:05.543554 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:10:05.543535    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96\": container with ID starting with d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96 not found: ID does not exist" containerID="d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96"
Oct 31 00:10:05.543630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.543562    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96"} err="failed to get container status \"d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96\": rpc error: code = NotFound desc = could not find container \"d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96\": container with ID starting with d1124f2efe8f1fb51472a20da9f2258f2d74fc0a8027369ee612fa44ff8fed96 not found: ID does not exist"
Oct 31 00:10:05.557250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.557225    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdf75"]
Oct 31 00:10:05.564123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:05.564103    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdf75"]
Oct 31 00:10:06.193526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:06.193492    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3ba51b-28cf-4496-a719-857d9e2e3763" path="/var/lib/kubelet/pods/7d3ba51b-28cf-4496-a719-857d9e2e3763/volumes"
Oct 31 00:10:06.384725 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:10:06.384689    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:10:35.830180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.830147    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r7nwt"]
Oct 31 00:10:35.830679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.830636    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7d3ba51b-28cf-4496-a719-857d9e2e3763" containerName="manager"
Oct 31 00:10:35.830679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.830657    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3ba51b-28cf-4496-a719-857d9e2e3763" containerName="manager"
Oct 31 00:10:35.830807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.830795    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3ba51b-28cf-4496-a719-857d9e2e3763" containerName="manager"
Oct 31 00:10:35.836912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.836897    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:35.840676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.840651    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r7nwt"]
Oct 31 00:10:35.849940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.849921    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gzz8j\""
Oct 31 00:10:35.906115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.906087    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r7nwt"]
Oct 31 00:10:35.906285 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:10:35.906268    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j7g6r], unattached volumes=[], failed to process volumes=[kube-api-access-j7g6r]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-r7nwt" podUID="4bb38091-9b38-412d-8064-c87103f23e0a"
Oct 31 00:10:35.946901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.946874    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8cgxs"]
Oct 31 00:10:35.950177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.950161    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:35.958870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.958830    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 00:10:35.960788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.960761    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8cgxs"]
Oct 31 00:10:35.992296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:35.992274    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7g6r\" (UniqueName: \"kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r\") pod \"authorino-6f6944bcdd-r7nwt\" (UID: \"4bb38091-9b38-412d-8064-c87103f23e0a\") " pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:36.093075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.093000    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gp8\" (UniqueName: \"kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8\") pod \"authorino-6889cbd8c4-8cgxs\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") " pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.093075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.093055    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7g6r\" (UniqueName: \"kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r\") pod \"authorino-6f6944bcdd-r7nwt\" (UID: \"4bb38091-9b38-412d-8064-c87103f23e0a\") " pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:36.093239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.093158    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert\") pod \"authorino-6889cbd8c4-8cgxs\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") " pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.106802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.106771    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7g6r\" (UniqueName: \"kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r\") pod \"authorino-6f6944bcdd-r7nwt\" (UID: \"4bb38091-9b38-412d-8064-c87103f23e0a\") " pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:36.193632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.193592    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gp8\" (UniqueName: \"kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8\") pod \"authorino-6889cbd8c4-8cgxs\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") " pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.193783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.193662    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert\") pod \"authorino-6889cbd8c4-8cgxs\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") " pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.196232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.196211    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert\") pod \"authorino-6889cbd8c4-8cgxs\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") " pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.203353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.203330    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8cgxs"]
Oct 31 00:10:36.203609 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:10:36.203584    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-w7gp8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-8cgxs" podUID="0b86e81b-322a-443e-9eee-081a4bd8c3d1"
Oct 31 00:10:36.210175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.210155    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gp8\" (UniqueName: \"kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8\") pod \"authorino-6889cbd8c4-8cgxs\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") " pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.233499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.233477    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c6964c754-7687k"]
Oct 31 00:10:36.236993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.236979    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.244227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.244208    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c6964c754-7687k"]
Oct 31 00:10:36.385788 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:10:36.385757    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:10:36.395192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.395172    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert\") pod \"authorino-c6964c754-7687k\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") " pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.395329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.395309    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8bn\" (UniqueName: \"kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn\") pod \"authorino-c6964c754-7687k\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") " pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.495760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.495729    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8bn\" (UniqueName: \"kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn\") pod \"authorino-c6964c754-7687k\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") " pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.495875 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.495768    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert\") pod \"authorino-c6964c754-7687k\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") " pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.498240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.498220    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert\") pod \"authorino-c6964c754-7687k\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") " pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.514359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.514336    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8bn\" (UniqueName: \"kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn\") pod \"authorino-c6964c754-7687k\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") " pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.564181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.564159    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:10:36.679968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.679903    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.680082 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.679904    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:36.698106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.698075    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:36.701415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.701398    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:36.798170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.798146    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8\" (UniqueName: \"kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8\") pod \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") "
Oct 31 00:10:36.798280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.798180    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert\" (UniqueName: \"kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert\") pod \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\" (UID: \"0b86e81b-322a-443e-9eee-081a4bd8c3d1\") "
Oct 31 00:10:36.798280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.798201    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r\" (UniqueName: \"kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r\") pod \"4bb38091-9b38-412d-8064-c87103f23e0a\" (UID: \"4bb38091-9b38-412d-8064-c87103f23e0a\") "
Oct 31 00:10:36.800511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.800485    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert" pod "0b86e81b-322a-443e-9eee-081a4bd8c3d1" (UID: "0b86e81b-322a-443e-9eee-081a4bd8c3d1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:10:36.800602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.800532    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r" pod "4bb38091-9b38-412d-8064-c87103f23e0a" (UID: "4bb38091-9b38-412d-8064-c87103f23e0a"). InnerVolumeSpecName "kube-api-access-j7g6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:10:36.800602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.800550    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8" pod "0b86e81b-322a-443e-9eee-081a4bd8c3d1" (UID: "0b86e81b-322a-443e-9eee-081a4bd8c3d1"). InnerVolumeSpecName "kube-api-access-w7gp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:10:36.835374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.835351    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c6964c754-7687k"]
Oct 31 00:10:36.836127 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:10:36.836106    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode80a2d2a_7776_4923_96a2_9b392618c53e.slice/crio-87cfddb803b8ec95199b4e77cc674f4fd64f20a416238e209469e2e365dfc1dd WatchSource:0}: Error finding container 87cfddb803b8ec95199b4e77cc674f4fd64f20a416238e209469e2e365dfc1dd: Status 404 returned error can't find the container with id 87cfddb803b8ec95199b4e77cc674f4fd64f20a416238e209469e2e365dfc1dd
Oct 31 00:10:36.899832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.899804    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gp8\" (UniqueName: \"kubernetes.io/projected/0b86e81b-322a-443e-9eee-081a4bd8c3d1-kube-api-access-w7gp8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:36.899832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.899825    5078 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b86e81b-322a-443e-9eee-081a4bd8c3d1-tls-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:36.899965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:36.899836    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7g6r\" (UniqueName: \"kubernetes.io/projected/4bb38091-9b38-412d-8064-c87103f23e0a-kube-api-access-j7g6r\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:37.686189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.686157    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6964c754-7687k" event={"ID":"e80a2d2a-7776-4923-96a2-9b392618c53e","Type":"ContainerStarted","Data":"87cfddb803b8ec95199b4e77cc674f4fd64f20a416238e209469e2e365dfc1dd"}
Oct 31 00:10:37.686354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.686197    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r7nwt"
Oct 31 00:10:37.686354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.686201    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8cgxs"
Oct 31 00:10:37.725177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.725154    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8cgxs"]
Oct 31 00:10:37.732680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.732657    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8cgxs"]
Oct 31 00:10:37.758030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.758000    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r7nwt"]
Oct 31 00:10:37.763100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:37.763076    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r7nwt"]
Oct 31 00:10:38.193500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:38.193448    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b86e81b-322a-443e-9eee-081a4bd8c3d1" path="/var/lib/kubelet/pods/0b86e81b-322a-443e-9eee-081a4bd8c3d1/volumes"
Oct 31 00:10:38.193854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:38.193796    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb38091-9b38-412d-8064-c87103f23e0a" path="/var/lib/kubelet/pods/4bb38091-9b38-412d-8064-c87103f23e0a/volumes"
Oct 31 00:10:38.693775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:38.693740    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6964c754-7687k" event={"ID":"e80a2d2a-7776-4923-96a2-9b392618c53e","Type":"ContainerStarted","Data":"35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839"}
Oct 31 00:10:38.716330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:10:38.716272    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c6964c754-7687k" podStartSLOduration=1.385349266 podStartE2EDuration="2.716257502s" totalImagesPullingTime="1.330908236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:38.715422177 +0000 UTC m=+1006.661792894" watchObservedRunningTime="2026-10-31 00:10:38.716257502 +0000 UTC m=+1006.662628218"
Oct 31 00:11:02.982183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:02.982107    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"]
Oct 31 00:11:02.986841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:02.986822    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:02.996231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:02.996207    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p7q4d\""
Oct 31 00:11:02.998106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:02.998076    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"]
Oct 31 00:11:03.007244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.007207    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"]
Oct 31 00:11:03.011416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.011399    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.036341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.036318    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"]
Oct 31 00:11:03.112290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112259    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.112290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112295    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.112567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112318    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.112567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112335    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.112567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112399    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.112567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112454    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zn9\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.112567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112562    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwbg\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.112788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112592    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.112788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112623    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.112788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112665    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.112788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112766    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.112788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112785    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.113007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112802    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.113007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112876    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.113007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112905    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.113007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112931    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.113007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.112991    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.113179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.113014    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214031    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214065    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214082    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214115    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214223    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214257    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214296    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214326    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214415    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214492    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214521    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214526    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214617    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214623    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214664    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.214690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214692    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214701    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214722    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214748    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214771    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zn9\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwbg\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214842    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214933    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.214994    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.215074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.215083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.215106    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.215595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.215286    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.217061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.217040    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.217061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.217050    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.217395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.217377    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.217576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.217560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.230597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.230567    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.230974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.230950    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.231109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.231092    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zn9\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-fstrj\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.231275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.231259    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwbg\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-f8j9v\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.300627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.300571    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:03.324001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.323980    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:03.579044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.579017    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"]
Oct 31 00:11:03.581211 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:03.581176    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfb2dac_d2ee_49e2_8c34_71bb157b0885.slice/crio-88953ccaafa6d0d6998a4b1745626547ef417810b003affc95b5f4a053acb83e WatchSource:0}: Error finding container 88953ccaafa6d0d6998a4b1745626547ef417810b003affc95b5f4a053acb83e: Status 404 returned error can't find the container with id 88953ccaafa6d0d6998a4b1745626547ef417810b003affc95b5f4a053acb83e
Oct 31 00:11:03.583397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.583365    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:03.583518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.583440    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:03.583518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.583501    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:03.610246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.610221    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"]
Oct 31 00:11:03.611291 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:03.611264    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f1a6d4_f73a_437d_8f92_5e027d6f0f96.slice/crio-ae9abee5d9d09c9b8f4d73e220869ed0d05756c456c4f45fee4a345d55446be1 WatchSource:0}: Error finding container ae9abee5d9d09c9b8f4d73e220869ed0d05756c456c4f45fee4a345d55446be1: Status 404 returned error can't find the container with id ae9abee5d9d09c9b8f4d73e220869ed0d05756c456c4f45fee4a345d55446be1
Oct 31 00:11:03.613456 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.613432    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:03.613528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.613518    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:03.613561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.613545    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:03.803049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.803001    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" event={"ID":"5bfb2dac-d2ee-49e2-8c34-71bb157b0885","Type":"ContainerStarted","Data":"a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6"}
Oct 31 00:11:03.803221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.803092    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" event={"ID":"5bfb2dac-d2ee-49e2-8c34-71bb157b0885","Type":"ContainerStarted","Data":"88953ccaafa6d0d6998a4b1745626547ef417810b003affc95b5f4a053acb83e"}
Oct 31 00:11:03.805050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.805022    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" event={"ID":"47f1a6d4-f73a-437d-8f92-5e027d6f0f96","Type":"ContainerStarted","Data":"0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a"}
Oct 31 00:11:03.805178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.805075    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" event={"ID":"47f1a6d4-f73a-437d-8f92-5e027d6f0f96","Type":"ContainerStarted","Data":"ae9abee5d9d09c9b8f4d73e220869ed0d05756c456c4f45fee4a345d55446be1"}
Oct 31 00:11:03.827971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.827927    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podStartSLOduration=1.8279157499999998 podStartE2EDuration="1.82791575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:03.827127278 +0000 UTC m=+1031.773497996" watchObservedRunningTime="2026-10-31 00:11:03.82791575 +0000 UTC m=+1031.774286466"
Oct 31 00:11:03.856213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:03.856122    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podStartSLOduration=1.856105702 podStartE2EDuration="1.856105702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:03.854125366 +0000 UTC m=+1031.800496083" watchObservedRunningTime="2026-10-31 00:11:03.856105702 +0000 UTC m=+1031.802476420"
Oct 31 00:11:04.307547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:04.307512    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:04.308228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:04.308202    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 31 00:11:04.308406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:04.308383    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 31 00:11:04.324516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:04.324485    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:04.325994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:04.325969    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:11:04.326107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:04.326018    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:11:05.301983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:05.301944    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 31 00:11:05.302273 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:05.302243    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 31 00:11:05.324818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:05.324788    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:11:05.325159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:05.324826    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:11:06.307489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:06.306020    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 31 00:11:06.307489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:06.306131    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 31 00:11:06.324601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:06.324571    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:11:06.324964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:06.324620    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:11:06.433151 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:11:06.433120    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:11:07.305488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:07.304912    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 31 00:11:07.305488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:07.304974    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 31 00:11:07.324711 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:07.324545    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:11:07.324711 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:07.324605    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:11:08.314489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:08.313987    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 31 00:11:08.314489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:08.314057    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 31 00:11:08.326015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:08.325989    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:11:08.326227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:08.326209    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:11:09.303490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:09.301831    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 31 00:11:09.303490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:09.301935    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 31 00:11:09.325274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:09.325243    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:11:09.325796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:09.325771    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:11:10.305020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:10.304988    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:10.305265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:10.305243    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:10.305796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:10.305778    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:11:10.328204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:10.328184    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:10.328560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:10.328441    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:10.329040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:10.329022    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:11:18.109504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.109454    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"]
Oct 31 00:11:18.113351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.113332    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.139564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.139539    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"]
Oct 31 00:11:18.259506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259455    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjzq\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259511    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259595    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259621    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259643    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259688    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259724    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259744    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.259903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.259771    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.360830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360771    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.360830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360797    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.360830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360817    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360838    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360853    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360868    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360892    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.360975    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjzq\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.361002    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.361222    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.361260    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.361348    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.361379    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.361690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.361671    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.363280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.363258    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.363642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.363624    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.375794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.375775    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjzq\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.375946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.375930    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5x24b\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.426413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.426390    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:18.702786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.702764    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"]
Oct 31 00:11:18.704448 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:18.704422    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1437fb_5c91_492e_ae01_54983042358a.slice/crio-d578b9cad7fe3136e86eb3048088f9cee234cc945d88d2720255c61ce0ff1827 WatchSource:0}: Error finding container d578b9cad7fe3136e86eb3048088f9cee234cc945d88d2720255c61ce0ff1827: Status 404 returned error can't find the container with id d578b9cad7fe3136e86eb3048088f9cee234cc945d88d2720255c61ce0ff1827
Oct 31 00:11:18.706635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.706605    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:18.706715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.706687    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:18.706754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.706714    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:11:18.892321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.892275    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" event={"ID":"0c1437fb-5c91-492e-ae01-54983042358a","Type":"ContainerStarted","Data":"c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706"}
Oct 31 00:11:18.892543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.892347    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" event={"ID":"0c1437fb-5c91-492e-ae01-54983042358a","Type":"ContainerStarted","Data":"d578b9cad7fe3136e86eb3048088f9cee234cc945d88d2720255c61ce0ff1827"}
Oct 31 00:11:18.952115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:18.952005    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podStartSLOduration=0.951990501 podStartE2EDuration="951.990501ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:18.949673862 +0000 UTC m=+1046.896044595" watchObservedRunningTime="2026-10-31 00:11:18.951990501 +0000 UTC m=+1046.898361215"
Oct 31 00:11:19.426970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:19.426928    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:19.428574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:19.428550    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:11:19.428694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:19.428595    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:11:20.427310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:20.427271    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:11:20.427738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:20.427321    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:11:21.427732 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:21.427699    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:11:21.428140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:21.427750    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:11:22.426806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:22.426772    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:11:22.426988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:22.426823    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:11:23.426807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:23.426770    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:11:23.427200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:23.426823    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:11:24.426959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:24.426926    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:11:24.427346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:24.426974    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:11:25.431785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:25.431753    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:25.432254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:25.432109    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:25.432789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:25.432770    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:11:34.567920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.567890    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"]
Oct 31 00:11:34.571814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.571795    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.578663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.578629    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 00:11:34.586279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.586257    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"]
Oct 31 00:11:34.594003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.593957    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wc2gc\""
Oct 31 00:11:34.594390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.594360    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 00:11:34.634643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.634601    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.634643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.634648    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.634827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.634671    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.634827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.634753    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kg5\" (UniqueName: \"kubernetes.io/projected/7aef86cd-1369-4407-a6a7-4febffe6d133-kube-api-access-m6kg5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.634892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.634836    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aef86cd-1369-4407-a6a7-4febffe6d133-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.634892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.634871    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.682908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.682882    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 00:11:34.735691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.735637    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aef86cd-1369-4407-a6a7-4febffe6d133-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.735885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.735711    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.735885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.735781    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.735885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.735812    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.735885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.735850    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.736094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.735900    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kg5\" (UniqueName: \"kubernetes.io/projected/7aef86cd-1369-4407-a6a7-4febffe6d133-kube-api-access-m6kg5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.736271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.736238    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.736574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.736552    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.737105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.737043    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.740159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.740118    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aef86cd-1369-4407-a6a7-4febffe6d133-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.740298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.740274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aef86cd-1369-4407-a6a7-4febffe6d133-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.755327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.755299    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kg5\" (UniqueName: \"kubernetes.io/projected/7aef86cd-1369-4407-a6a7-4febffe6d133-kube-api-access-m6kg5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs\" (UID: \"7aef86cd-1369-4407-a6a7-4febffe6d133\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:34.886382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:34.886349    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:35.165693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:35.165664    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"]
Oct 31 00:11:35.167260 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:35.167237    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aef86cd_1369_4407_a6a7_4febffe6d133.slice/crio-f4748dabb26e52ee653650e03a6e616a88ab4c4fc0473520f5f9b612b9631db9 WatchSource:0}: Error finding container f4748dabb26e52ee653650e03a6e616a88ab4c4fc0473520f5f9b612b9631db9: Status 404 returned error can't find the container with id f4748dabb26e52ee653650e03a6e616a88ab4c4fc0473520f5f9b612b9631db9
Oct 31 00:11:36.002811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.002771    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" event={"ID":"7aef86cd-1369-4407-a6a7-4febffe6d133","Type":"ContainerStarted","Data":"f4748dabb26e52ee653650e03a6e616a88ab4c4fc0473520f5f9b612b9631db9"}
Oct 31 00:11:36.397041 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:11:36.396985    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:11:36.832751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.832668    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"]
Oct 31 00:11:36.838025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.837999    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:36.846085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.846050    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"]
Oct 31 00:11:36.855946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.855919    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 00:11:36.900696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.900664    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9p9\" (UniqueName: \"kubernetes.io/projected/ee668f0d-fda1-4731-83cf-954fa3780710-kube-api-access-7c9p9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:36.900876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.900713    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:36.900876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.900811    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:36.900876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.900840    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee668f0d-fda1-4731-83cf-954fa3780710-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:36.901013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.900992    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:36.901013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:36.901007    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.001613    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.001662    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.001718    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9p9\" (UniqueName: \"kubernetes.io/projected/ee668f0d-fda1-4731-83cf-954fa3780710-kube-api-access-7c9p9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.001757    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.001827    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.001852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee668f0d-fda1-4731-83cf-954fa3780710-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.002522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.002217    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.005863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.005836    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.005974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.005907    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.010567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.010519    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee668f0d-fda1-4731-83cf-954fa3780710-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.010919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.010853    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee668f0d-fda1-4731-83cf-954fa3780710-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.016222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.015963    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9p9\" (UniqueName: \"kubernetes.io/projected/ee668f0d-fda1-4731-83cf-954fa3780710-kube-api-access-7c9p9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk\" (UID: \"ee668f0d-fda1-4731-83cf-954fa3780710\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.158361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.158335    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:37.441672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:37.441644    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"]
Oct 31 00:11:37.443089 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:37.443062    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee668f0d_fda1_4731_83cf_954fa3780710.slice/crio-e0c34ff19eb3ea3d78689e9072c8893bf3bb428526e890120c4b672923c3e0ab WatchSource:0}: Error finding container e0c34ff19eb3ea3d78689e9072c8893bf3bb428526e890120c4b672923c3e0ab: Status 404 returned error can't find the container with id e0c34ff19eb3ea3d78689e9072c8893bf3bb428526e890120c4b672923c3e0ab
Oct 31 00:11:38.024537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.023496    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" event={"ID":"ee668f0d-fda1-4731-83cf-954fa3780710","Type":"ContainerStarted","Data":"09a34d143f903c0ec1f18b1bbea125021e96405a6202cc461ce009fc4d6eff26"}
Oct 31 00:11:38.024537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.023598    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" event={"ID":"ee668f0d-fda1-4731-83cf-954fa3780710","Type":"ContainerStarted","Data":"e0c34ff19eb3ea3d78689e9072c8893bf3bb428526e890120c4b672923c3e0ab"}
Oct 31 00:11:38.032483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.026770    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" event={"ID":"7aef86cd-1369-4407-a6a7-4febffe6d133","Type":"ContainerStarted","Data":"9ecf3d1ac64766955b7b35f069041ec5f424519a44450a69404f6c8b9326caa1"}
Oct 31 00:11:38.076208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.076146    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podStartSLOduration=2.1300654039999998 podStartE2EDuration="4.076127925s" totalImagesPullingTime="1.946062521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:38.074229 +0000 UTC m=+1066.020599717" watchObservedRunningTime="2026-10-31 00:11:38.076127925 +0000 UTC m=+1066.022498643"
Oct 31 00:11:38.076561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.076523    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podStartSLOduration=1.8074596029999999 podStartE2EDuration="2.07651437s" totalImagesPullingTime="269.054767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:38.048878883 +0000 UTC m=+1065.995249599" watchObservedRunningTime="2026-10-31 00:11:38.07651437 +0000 UTC m=+1066.022885088"
Oct 31 00:11:38.158495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.158438    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:38.890489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:38.889548    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:40.637748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.637713    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"]
Oct 31 00:11:40.642296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.642272    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.649588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.649562    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 00:11:40.659701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.659674    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"]
Oct 31 00:11:40.753264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.753218    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.753481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.753276    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.753481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.753393    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6821bd-245e-4dd1-a279-d34358f5d3a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.753611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.753592    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.753666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.753621    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksv8\" (UniqueName: \"kubernetes.io/projected/dc6821bd-245e-4dd1-a279-d34358f5d3a1-kube-api-access-2ksv8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.754395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.754367    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.855713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.855654    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.855713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.855687    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksv8\" (UniqueName: \"kubernetes.io/projected/dc6821bd-245e-4dd1-a279-d34358f5d3a1-kube-api-access-2ksv8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.855713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.855719    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.856045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.855749    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.856045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.855835    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.856045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.855894    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6821bd-245e-4dd1-a279-d34358f5d3a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.856289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.856258    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.856361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.856271    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.856478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.856444    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.858283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.858257    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6821bd-245e-4dd1-a279-d34358f5d3a1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.858719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.858695    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6821bd-245e-4dd1-a279-d34358f5d3a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.873175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.873147    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksv8\" (UniqueName: \"kubernetes.io/projected/dc6821bd-245e-4dd1-a279-d34358f5d3a1-kube-api-access-2ksv8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-67244\" (UID: \"dc6821bd-245e-4dd1-a279-d34358f5d3a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:40.960137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:40.960036    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:41.285645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:41.285611    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"]
Oct 31 00:11:42.127291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:42.127174    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" event={"ID":"dc6821bd-245e-4dd1-a279-d34358f5d3a1","Type":"ContainerStarted","Data":"151f699f4af83b6931a80becf4e43074812d43e2899b53fdcf8c484696307a2b"}
Oct 31 00:11:42.127972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:42.127867    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" event={"ID":"dc6821bd-245e-4dd1-a279-d34358f5d3a1","Type":"ContainerStarted","Data":"fb0ac7a7c5c73a6b2341f0126dad8ecf8fdde934405f0f458ba64ea521adf9a0"}
Oct 31 00:11:42.166705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:42.166625    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podStartSLOduration=1.693333332 podStartE2EDuration="2.166604182s" totalImagesPullingTime="473.27085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:42.162322079 +0000 UTC m=+1070.108692797" watchObservedRunningTime="2026-10-31 00:11:42.166604182 +0000 UTC m=+1070.112974900"
Oct 31 00:11:42.960997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:42.960962    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:43.391759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.389908    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"]
Oct 31 00:11:43.397735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.397709    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.428841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.428807    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"]
Oct 31 00:11:43.440452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.440056    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 00:11:43.554061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.554015    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3307b72b-996a-4b66-823c-e4347c25e18f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.554413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.554385    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.554590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.554569    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gz5h\" (UniqueName: \"kubernetes.io/projected/3307b72b-996a-4b66-823c-e4347c25e18f-kube-api-access-4gz5h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.554768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.554751    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.554938 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.554921    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.555072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.555055    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.658537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.657943    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.658537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.657996    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gz5h\" (UniqueName: \"kubernetes.io/projected/3307b72b-996a-4b66-823c-e4347c25e18f-kube-api-access-4gz5h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.658537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.658062    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.658537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.658139    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.658537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.658165    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.658537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.658223    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3307b72b-996a-4b66-823c-e4347c25e18f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.677485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.662859    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.677485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.663090    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.679342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.678827    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.688534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.683196    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3307b72b-996a-4b66-823c-e4347c25e18f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.697482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.695745    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3307b72b-996a-4b66-823c-e4347c25e18f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.699949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.699920    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gz5h\" (UniqueName: \"kubernetes.io/projected/3307b72b-996a-4b66-823c-e4347c25e18f-kube-api-access-4gz5h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv\" (UID: \"3307b72b-996a-4b66-823c-e4347c25e18f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:43.750487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:43.748613    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:44.244554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:44.237570    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"]
Oct 31 00:11:44.265610 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:44.265582    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3307b72b_996a_4b66_823c_e4347c25e18f.slice/crio-d1a6128abf296c04c76d641f44e4f23b80ef6deb9f52e7e6813cbfae7bac7c37 WatchSource:0}: Error finding container d1a6128abf296c04c76d641f44e4f23b80ef6deb9f52e7e6813cbfae7bac7c37: Status 404 returned error can't find the container with id d1a6128abf296c04c76d641f44e4f23b80ef6deb9f52e7e6813cbfae7bac7c37
Oct 31 00:11:44.894896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:44.894872    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:44.964590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:44.964513    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:45.168617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:45.168579    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" event={"ID":"3307b72b-996a-4b66-823c-e4347c25e18f","Type":"ContainerStarted","Data":"c0b65f25eb66aa6b9862fff64d00c9d6525a95384a3e52ca8cadf6fb34d12cec"}
Oct 31 00:11:45.168833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:45.168659    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" event={"ID":"3307b72b-996a-4b66-823c-e4347c25e18f","Type":"ContainerStarted","Data":"d1a6128abf296c04c76d641f44e4f23b80ef6deb9f52e7e6813cbfae7bac7c37"}
Oct 31 00:11:45.188529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:45.188496    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:11:45.193946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:45.193892    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podStartSLOduration=1.802928236 podStartE2EDuration="2.193877445s" totalImagesPullingTime="390.949209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:45.192855315 +0000 UTC m=+1073.139226035" watchObservedRunningTime="2026-10-31 00:11:45.193877445 +0000 UTC m=+1073.140248164"
Oct 31 00:11:45.749955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:45.749917    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:47.160497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:47.159561    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:47.180617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:47.180593    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:47.237771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:47.237743    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:11:50.961105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:50.961066    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:50.990795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:50.990766    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:51.362766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:51.362681    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:11:53.749405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:53.749368    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:53.764109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:53.764080    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:54.368777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:54.368750    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:11:56.784415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:56.784380    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58544c6c77-l9nc7"]
Oct 31 00:11:56.789792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:56.789762    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:56.800109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:56.800087    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58544c6c77-l9nc7"]
Oct 31 00:11:56.905374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:56.905342    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4m59\" (UniqueName: \"kubernetes.io/projected/fdac0d2e-793d-4fad-a635-528229ed9df1-kube-api-access-f4m59\") pod \"authorino-58544c6c77-l9nc7\" (UID: \"fdac0d2e-793d-4fad-a635-528229ed9df1\") " pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:56.905374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:56.905378    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fdac0d2e-793d-4fad-a635-528229ed9df1-tls-cert\") pod \"authorino-58544c6c77-l9nc7\" (UID: \"fdac0d2e-793d-4fad-a635-528229ed9df1\") " pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:57.006167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:57.006132    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4m59\" (UniqueName: \"kubernetes.io/projected/fdac0d2e-793d-4fad-a635-528229ed9df1-kube-api-access-f4m59\") pod \"authorino-58544c6c77-l9nc7\" (UID: \"fdac0d2e-793d-4fad-a635-528229ed9df1\") " pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:57.006344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:57.006177    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fdac0d2e-793d-4fad-a635-528229ed9df1-tls-cert\") pod \"authorino-58544c6c77-l9nc7\" (UID: \"fdac0d2e-793d-4fad-a635-528229ed9df1\") " pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:57.008776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:57.008750    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fdac0d2e-793d-4fad-a635-528229ed9df1-tls-cert\") pod \"authorino-58544c6c77-l9nc7\" (UID: \"fdac0d2e-793d-4fad-a635-528229ed9df1\") " pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:57.019865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:57.019837    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4m59\" (UniqueName: \"kubernetes.io/projected/fdac0d2e-793d-4fad-a635-528229ed9df1-kube-api-access-f4m59\") pod \"authorino-58544c6c77-l9nc7\" (UID: \"fdac0d2e-793d-4fad-a635-528229ed9df1\") " pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:57.102380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:57.102303    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58544c6c77-l9nc7"
Oct 31 00:11:57.377187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:57.377114    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58544c6c77-l9nc7"]
Oct 31 00:11:57.380800 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:11:57.380770    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdac0d2e_793d_4fad_a635_528229ed9df1.slice/crio-47fe3f21381630a4f13b9efc60f3730306e543082ce130cc09c5bcf4b18ef397 WatchSource:0}: Error finding container 47fe3f21381630a4f13b9efc60f3730306e543082ce130cc09c5bcf4b18ef397: Status 404 returned error can't find the container with id 47fe3f21381630a4f13b9efc60f3730306e543082ce130cc09c5bcf4b18ef397
Oct 31 00:11:58.384107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.384066    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58544c6c77-l9nc7" event={"ID":"fdac0d2e-793d-4fad-a635-528229ed9df1","Type":"ContainerStarted","Data":"7886a9ba0524409c89c9c22f4337bf533bfbdcfef0982ab7f873967d5e9a903e"}
Oct 31 00:11:58.384518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.384124    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58544c6c77-l9nc7" event={"ID":"fdac0d2e-793d-4fad-a635-528229ed9df1","Type":"ContainerStarted","Data":"47fe3f21381630a4f13b9efc60f3730306e543082ce130cc09c5bcf4b18ef397"}
Oct 31 00:11:58.406642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.406590    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58544c6c77-l9nc7" podStartSLOduration=1.526627198 podStartE2EDuration="2.406575517s" totalImagesPullingTime="879.948319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:58.40353677 +0000 UTC m=+1086.349907495" watchObservedRunningTime="2026-10-31 00:11:58.406575517 +0000 UTC m=+1086.352946240"
Oct 31 00:11:58.463339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.463304    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c6964c754-7687k"]
Oct 31 00:11:58.463573 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.463542    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c6964c754-7687k" podUID="e80a2d2a-7776-4923-96a2-9b392618c53e" containerName="authorino" containerID="cri-o://35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839" gracePeriod=30
Oct 31 00:11:58.722509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.722446    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:11:58.825183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.825151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert\" (UniqueName: \"kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert\") pod \"e80a2d2a-7776-4923-96a2-9b392618c53e\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") "
Oct 31 00:11:58.825380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.825253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn\" (UniqueName: \"kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn\") pod \"e80a2d2a-7776-4923-96a2-9b392618c53e\" (UID: \"e80a2d2a-7776-4923-96a2-9b392618c53e\") "
Oct 31 00:11:58.827577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.827544    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn" pod "e80a2d2a-7776-4923-96a2-9b392618c53e" (UID: "e80a2d2a-7776-4923-96a2-9b392618c53e"). InnerVolumeSpecName "kube-api-access-6m8bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:58.843124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.843043    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert" pod "e80a2d2a-7776-4923-96a2-9b392618c53e" (UID: "e80a2d2a-7776-4923-96a2-9b392618c53e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:11:58.926986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.926953    5078 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e80a2d2a-7776-4923-96a2-9b392618c53e-tls-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:58.926986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:58.926980    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8bn\" (UniqueName: \"kubernetes.io/projected/e80a2d2a-7776-4923-96a2-9b392618c53e-kube-api-access-6m8bn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:59.247860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.247820    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"]
Oct 31 00:11:59.248427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.248407    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e80a2d2a-7776-4923-96a2-9b392618c53e" containerName="authorino"
Oct 31 00:11:59.248543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.248430    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80a2d2a-7776-4923-96a2-9b392618c53e" containerName="authorino"
Oct 31 00:11:59.248610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.248578    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80a2d2a-7776-4923-96a2-9b392618c53e" containerName="authorino"
Oct 31 00:11:59.251930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.251907    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.267908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.267880    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 00:11:59.268054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.267986    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 00:11:59.268134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.268101    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-kbkl9\""
Oct 31 00:11:59.274424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.274399    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"]
Oct 31 00:11:59.391091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.391051    5078 generic.go:356] "Generic (PLEG): container finished" podID="e80a2d2a-7776-4923-96a2-9b392618c53e" containerID="35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839" exitCode=0
Oct 31 00:11:59.391596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.391098    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6964c754-7687k"
Oct 31 00:11:59.391596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.391135    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6964c754-7687k" event={"ID":"e80a2d2a-7776-4923-96a2-9b392618c53e","Type":"ContainerDied","Data":"35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839"}
Oct 31 00:11:59.391596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.391177    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6964c754-7687k" event={"ID":"e80a2d2a-7776-4923-96a2-9b392618c53e","Type":"ContainerDied","Data":"87cfddb803b8ec95199b4e77cc674f4fd64f20a416238e209469e2e365dfc1dd"}
Oct 31 00:11:59.391596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.391198    5078 scope.go:122] "RemoveContainer" containerID="35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839"
Oct 31 00:11:59.402299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.402281    5078 scope.go:122] "RemoveContainer" containerID="35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839"
Oct 31 00:11:59.402665 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:11:59.402648    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839\": container with ID starting with 35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839 not found: ID does not exist" containerID="35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839"
Oct 31 00:11:59.402728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.402675    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839"} err="failed to get container status \"35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839\": rpc error: code = NotFound desc = could not find container \"35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839\": container with ID starting with 35d178b6cb5058288ffab78ded0439f5529bf3b0eb484ef7cdd882f24e063839 not found: ID does not exist"
Oct 31 00:11:59.426485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.425935    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c6964c754-7687k"]
Oct 31 00:11:59.436232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.436038    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-dbg69\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.436232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.436075    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c6964c754-7687k"]
Oct 31 00:11:59.436232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.436203    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwpc\" (UniqueName: \"kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc\") pod \"ai-gateway-controller-6cf7bcbfb6-dbg69\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.537020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.536925    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwpc\" (UniqueName: \"kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc\") pod \"ai-gateway-controller-6cf7bcbfb6-dbg69\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.537020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.537011    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-dbg69\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.539700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.539671    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-dbg69\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.552419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.552390    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwpc\" (UniqueName: \"kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc\") pod \"ai-gateway-controller-6cf7bcbfb6-dbg69\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.570257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.570235    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:11:59.875026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:11:59.874999    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"]
Oct 31 00:12:00.195014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:00.194982    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80a2d2a-7776-4923-96a2-9b392618c53e" path="/var/lib/kubelet/pods/e80a2d2a-7776-4923-96a2-9b392618c53e/volumes"
Oct 31 00:12:00.399372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:00.399341    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69" event={"ID":"42bb30ca-a5a0-4f68-9a5c-081f4d406bea","Type":"ContainerStarted","Data":"eadf0e1324dcf2cbe069b6f528a096e5b52fd621945e93dc54882ca55cacd2d1"}
Oct 31 00:12:03.417717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:03.417678    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69" event={"ID":"42bb30ca-a5a0-4f68-9a5c-081f4d406bea","Type":"ContainerStarted","Data":"c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9"}
Oct 31 00:12:03.418199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:03.417836    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:12:03.439385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:03.439318    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69" podStartSLOduration=1.8895052799999998 podStartE2EDuration="4.439302253s" totalImagesPullingTime="2.549796973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:03.438310943 +0000 UTC m=+1091.384681659" watchObservedRunningTime="2026-10-31 00:12:03.439302253 +0000 UTC m=+1091.385672969"
Oct 31 00:12:04.441709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.441678    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"]
Oct 31 00:12:04.445982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.445957    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.460112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.460086    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"]
Oct 31 00:12:04.461954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.461819    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pxrsw\""
Oct 31 00:12:04.461954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.461859    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 00:12:04.500554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.500518    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxw2\" (UniqueName: \"kubernetes.io/projected/fe713694-148e-4e08-bc8a-becc4f002df5-kube-api-access-vvxw2\") pod \"payload-pre-processing-7b98bb678c-c2w2g\" (UID: \"fe713694-148e-4e08-bc8a-becc4f002df5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.500732 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.500669    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe713694-148e-4e08-bc8a-becc4f002df5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-c2w2g\" (UID: \"fe713694-148e-4e08-bc8a-becc4f002df5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.601741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.601699    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxw2\" (UniqueName: \"kubernetes.io/projected/fe713694-148e-4e08-bc8a-becc4f002df5-kube-api-access-vvxw2\") pod \"payload-pre-processing-7b98bb678c-c2w2g\" (UID: \"fe713694-148e-4e08-bc8a-becc4f002df5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.601950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.601795    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe713694-148e-4e08-bc8a-becc4f002df5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-c2w2g\" (UID: \"fe713694-148e-4e08-bc8a-becc4f002df5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.602608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.602584    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe713694-148e-4e08-bc8a-becc4f002df5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-c2w2g\" (UID: \"fe713694-148e-4e08-bc8a-becc4f002df5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.622586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.622171    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxw2\" (UniqueName: \"kubernetes.io/projected/fe713694-148e-4e08-bc8a-becc4f002df5-kube-api-access-vvxw2\") pod \"payload-pre-processing-7b98bb678c-c2w2g\" (UID: \"fe713694-148e-4e08-bc8a-becc4f002df5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:04.760685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:04.760596    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:05.053906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:05.053872    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"]
Oct 31 00:12:05.054357 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:12:05.054323    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe713694_148e_4e08_bc8a_becc4f002df5.slice/crio-2bee1c566eeb49daab56e94b5c11484a8b5f73d62f355b87f17dc89e2372df05 WatchSource:0}: Error finding container 2bee1c566eeb49daab56e94b5c11484a8b5f73d62f355b87f17dc89e2372df05: Status 404 returned error can't find the container with id 2bee1c566eeb49daab56e94b5c11484a8b5f73d62f355b87f17dc89e2372df05
Oct 31 00:12:05.430088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:05.430050    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" event={"ID":"fe713694-148e-4e08-bc8a-becc4f002df5","Type":"ContainerStarted","Data":"2bee1c566eeb49daab56e94b5c11484a8b5f73d62f355b87f17dc89e2372df05"}
Oct 31 00:12:06.391922 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:12:06.391873    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:12:08.447767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:08.447727    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" event={"ID":"fe713694-148e-4e08-bc8a-becc4f002df5","Type":"ContainerStarted","Data":"07363445c622468b283b524ef8bca476d970190e0d5c82ccf0f063ad6ffa307e"}
Oct 31 00:12:08.448173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:08.447792    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:08.468765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:08.468721    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" podStartSLOduration=1.464982117 podStartE2EDuration="4.468708918s" totalImagesPullingTime="3.003726801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:08.467567728 +0000 UTC m=+1096.413938446" watchObservedRunningTime="2026-10-31 00:12:08.468708918 +0000 UTC m=+1096.415079633"
Oct 31 00:12:14.426217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.426187    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:12:14.704521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.704439    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86f7d849b4-vgljn"]
Oct 31 00:12:14.710926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.710905    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:14.718788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.718758    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86f7d849b4-vgljn"]
Oct 31 00:12:14.800581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.800555    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7j5\" (UniqueName: \"kubernetes.io/projected/6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152-kube-api-access-9l7j5\") pod \"ai-gateway-controller-86f7d849b4-vgljn\" (UID: \"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152\") " pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:14.800581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.800585    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152-metrics-certs\") pod \"ai-gateway-controller-86f7d849b4-vgljn\" (UID: \"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152\") " pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:14.901906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.901875    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7j5\" (UniqueName: \"kubernetes.io/projected/6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152-kube-api-access-9l7j5\") pod \"ai-gateway-controller-86f7d849b4-vgljn\" (UID: \"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152\") " pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:14.901906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.901905    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152-metrics-certs\") pod \"ai-gateway-controller-86f7d849b4-vgljn\" (UID: \"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152\") " pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:14.904551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.904531    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152-metrics-certs\") pod \"ai-gateway-controller-86f7d849b4-vgljn\" (UID: \"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152\") " pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:14.915382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:14.915364    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7j5\" (UniqueName: \"kubernetes.io/projected/6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152-kube-api-access-9l7j5\") pod \"ai-gateway-controller-86f7d849b4-vgljn\" (UID: \"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152\") " pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:15.030700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:15.030646    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:15.304209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:15.304178    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86f7d849b4-vgljn"]
Oct 31 00:12:15.306054 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:12:15.306025    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc5fbbb_ba8e_4b51_ba9c_d4efbba92152.slice/crio-b615725b4b4b14ae4a898f1d01af8ac233b0fb28b1286db10edf27b3064facbc WatchSource:0}: Error finding container b615725b4b4b14ae4a898f1d01af8ac233b0fb28b1286db10edf27b3064facbc: Status 404 returned error can't find the container with id b615725b4b4b14ae4a898f1d01af8ac233b0fb28b1286db10edf27b3064facbc
Oct 31 00:12:15.482716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:15.482681    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" event={"ID":"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152","Type":"ContainerStarted","Data":"b615725b4b4b14ae4a898f1d01af8ac233b0fb28b1286db10edf27b3064facbc"}
Oct 31 00:12:16.489533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:16.489490    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" event={"ID":"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152","Type":"ContainerStarted","Data":"775f4e59b730affd598dac624b3243c1b80a7ae6a20e2796c6aec75089940bec"}
Oct 31 00:12:16.489962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:16.489716    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:16.514497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:16.514439    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" podStartSLOduration=2.013347208 podStartE2EDuration="2.514425862s" totalImagesPullingTime="501.078654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:16.511086161 +0000 UTC m=+1104.457456879" watchObservedRunningTime="2026-10-31 00:12:16.514425862 +0000 UTC m=+1104.460796582"
Oct 31 00:12:19.456381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:19.456352    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:12:27.496586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.496555    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:12:27.547424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.547385    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"]
Oct 31 00:12:27.547912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.547854    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69" podUID="42bb30ca-a5a0-4f68-9a5c-081f4d406bea" containerName="manager" containerID="cri-o://c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9" gracePeriod=30
Oct 31 00:12:27.792391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.792366    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:12:27.926072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.926032    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc\" (UniqueName: \"kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc\") pod \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") "
Oct 31 00:12:27.926261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.926128    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs\") pod \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\" (UID: \"42bb30ca-a5a0-4f68-9a5c-081f4d406bea\") "
Oct 31 00:12:27.928448 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.928412    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs" pod "42bb30ca-a5a0-4f68-9a5c-081f4d406bea" (UID: "42bb30ca-a5a0-4f68-9a5c-081f4d406bea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:27.928667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:27.928645    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc" pod "42bb30ca-a5a0-4f68-9a5c-081f4d406bea" (UID: "42bb30ca-a5a0-4f68-9a5c-081f4d406bea"). InnerVolumeSpecName "kube-api-access-cfwpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:28.027825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.027759    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:28.027825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.027786    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwpc\" (UniqueName: \"kubernetes.io/projected/42bb30ca-a5a0-4f68-9a5c-081f4d406bea-kube-api-access-cfwpc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:28.554160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.554130    5078 generic.go:356] "Generic (PLEG): container finished" podID="42bb30ca-a5a0-4f68-9a5c-081f4d406bea" containerID="c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9" exitCode=0
Oct 31 00:12:28.554617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.554176    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"
Oct 31 00:12:28.554617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.554218    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69" event={"ID":"42bb30ca-a5a0-4f68-9a5c-081f4d406bea","Type":"ContainerDied","Data":"c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9"}
Oct 31 00:12:28.554617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.554267    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69" event={"ID":"42bb30ca-a5a0-4f68-9a5c-081f4d406bea","Type":"ContainerDied","Data":"eadf0e1324dcf2cbe069b6f528a096e5b52fd621945e93dc54882ca55cacd2d1"}
Oct 31 00:12:28.554617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.554290    5078 scope.go:122] "RemoveContainer" containerID="c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9"
Oct 31 00:12:28.562971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.562951    5078 scope.go:122] "RemoveContainer" containerID="c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9"
Oct 31 00:12:28.563239 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:12:28.563223    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9\": container with ID starting with c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9 not found: ID does not exist" containerID="c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9"
Oct 31 00:12:28.563311 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.563247    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9"} err="failed to get container status \"c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9\": rpc error: code = NotFound desc = could not find container \"c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9\": container with ID starting with c931ca24369b793ad541ac70de4dd8629dea0cf4d177d1a505083e7fa5fe13c9 not found: ID does not exist"
Oct 31 00:12:28.575790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.575766    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"]
Oct 31 00:12:28.580388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:28.580371    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-dbg69"]
Oct 31 00:12:30.193251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:12:30.193219    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42bb30ca-a5a0-4f68-9a5c-081f4d406bea" path="/var/lib/kubelet/pods/42bb30ca-a5a0-4f68-9a5c-081f4d406bea/volumes"
Oct 31 00:12:36.385136 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:12:36.385052    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:13:02.308874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.308838    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"]
Oct 31 00:13:02.309512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.309490    5078 cpu_manager.go:414] "containerMap: removing container" podUID="42bb30ca-a5a0-4f68-9a5c-081f4d406bea" containerName="manager"
Oct 31 00:13:02.309512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.309510    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42bb30ca-a5a0-4f68-9a5c-081f4d406bea" containerName="manager"
Oct 31 00:13:02.309710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.309614    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="42bb30ca-a5a0-4f68-9a5c-081f4d406bea" containerName="manager"
Oct 31 00:13:02.321952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.321914    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"]
Oct 31 00:13:02.322278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.322033    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.329148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.329122    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7c5da3-openshift-default-dockercfg-87p8b\""
Oct 31 00:13:02.405158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.405125    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"]
Oct 31 00:13:02.409604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.409581    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.426701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.426672    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"]
Oct 31 00:13:02.429871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.429846    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d91de6-openshift-default-dockercfg-p87px\""
Oct 31 00:13:02.446101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446058    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446121    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446138    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446155    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446218    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446242    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446269    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446333    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swscj\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.446443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.446364    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547154    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547329    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swscj\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547379    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547427    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4cbn\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547512    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547561    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547623    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547659    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.547700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547683    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547729    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547760    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547785    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547811    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547858    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547891    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547920    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547952    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.548145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.547980    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.549250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.549149    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.549443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.549422    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.549584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.549563    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.549746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.549726    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.549894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.549833    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.551573 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.551539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.552040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.552014    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.570995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.570925    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swscj\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.574805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.574779    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.628268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.628227    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"]
Oct 31 00:13:02.634120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.634097    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.643494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.639313    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:02.643494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.643234    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"]
Oct 31 00:13:02.643494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.640777    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c60826-openshift-default-dockercfg-v9d6d\""
Oct 31 00:13:02.650246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650221    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.650438 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650415    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.650717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650475    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.650717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650515    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.650717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650594    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4cbn\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.650717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650623    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.650717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650668    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.651045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650717    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.651045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.650754    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.651143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.651109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.651347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.651324    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.652011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.651854    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.652011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.651930    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.652432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.652409    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.654681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.654660    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.654748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.654680    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.674425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.674028    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4cbn\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.677352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.677321    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.727364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.727292    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.751859    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.751984    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752140    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752296    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752379    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752444    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752525    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kzp\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752618    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.753482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.752730    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.829454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.829374    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"]
Oct 31 00:13:02.834982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.834958    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.845288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.845256    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-19adb9-openshift-default-dockercfg-csw9z\""
Oct 31 00:13:02.849695 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.849670    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"]
Oct 31 00:13:02.856706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856678    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856750    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856787    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856815    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kzp\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856867    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856897    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856945    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.856973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.857041    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.857254    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.857726    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.857925    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.858772    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.860121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.859740    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.862420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.862378    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.870740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.868652    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.872988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.872970    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.874128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.874103    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kzp\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:02.938750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.938724    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"]
Oct 31 00:13:02.940350 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:02.940311    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd56ff0f_da7a_4de9_b2a0_125e471de6d2.slice/crio-9256a64ae94df3e36f50386e5596fec183a594cba6b1b7b8a9e95cf9dfb8693c WatchSource:0}: Error finding container 9256a64ae94df3e36f50386e5596fec183a594cba6b1b7b8a9e95cf9dfb8693c: Status 404 returned error can't find the container with id 9256a64ae94df3e36f50386e5596fec183a594cba6b1b7b8a9e95cf9dfb8693c
Oct 31 00:13:02.942940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.942911    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:02.943020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.942972    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:02.943020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.943001    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:02.958211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958180    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958249    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958416    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958502    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958536    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958627    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958664    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wj8\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.958813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.958712    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:02.960834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:02.960812    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:03.056966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.048095    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"]
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061640    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061721    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061818    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061863    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061896    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061926    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061958    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.061985    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wj8\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.062246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.062036    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.064768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.063389    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.064768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.063711    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.064768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.064242    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.064768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.064488    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.065595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.065328    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.065815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.065772    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.065910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.065865    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.065972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.065902    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.066981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.066957    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.077252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.074179    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.089556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.089445    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.101035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.101002    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wj8\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.156832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.156787    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:03.300115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.300079    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"]
Oct 31 00:13:03.312611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.312141    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.312611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.312221    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.312611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.312251    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.478142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.477636    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"]
Oct 31 00:13:03.478381 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:03.478270    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3aadd02_d322_476b_893b_2459b5f12808.slice/crio-0cc2068a8ef146ab0d3f454f808174c19ec117c8352c006923d48ff073d09fd7 WatchSource:0}: Error finding container 0cc2068a8ef146ab0d3f454f808174c19ec117c8352c006923d48ff073d09fd7: Status 404 returned error can't find the container with id 0cc2068a8ef146ab0d3f454f808174c19ec117c8352c006923d48ff073d09fd7
Oct 31 00:13:03.481924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.481890    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.482076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.481969    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.482076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.482014    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:03.730788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.730748    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" event={"ID":"48402b00-cb93-46fe-87c6-4e1437aca83c","Type":"ContainerStarted","Data":"558a54efa4466a8231c82bbe4ba518f3e330770b35d8ae092de99be0843a513b"}
Oct 31 00:13:03.730993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.730821    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" event={"ID":"48402b00-cb93-46fe-87c6-4e1437aca83c","Type":"ContainerStarted","Data":"142866db55d9d812f54c582c1544e4f27040541b10e194104209552827560600"}
Oct 31 00:13:03.732901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.732875    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" event={"ID":"f3aadd02-d322-476b-893b-2459b5f12808","Type":"ContainerStarted","Data":"f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"}
Oct 31 00:13:03.733043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.732924    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" event={"ID":"f3aadd02-d322-476b-893b-2459b5f12808","Type":"ContainerStarted","Data":"0cc2068a8ef146ab0d3f454f808174c19ec117c8352c006923d48ff073d09fd7"}
Oct 31 00:13:03.735037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.735015    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" event={"ID":"cd56ff0f-da7a-4de9-b2a0-125e471de6d2","Type":"ContainerStarted","Data":"edad74b1e307270b0d17d11a44f9221935caeb1c1df5382db1f63c3fd0f37305"}
Oct 31 00:13:03.735169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.735056    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" event={"ID":"cd56ff0f-da7a-4de9-b2a0-125e471de6d2","Type":"ContainerStarted","Data":"9256a64ae94df3e36f50386e5596fec183a594cba6b1b7b8a9e95cf9dfb8693c"}
Oct 31 00:13:03.737086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.737066    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" event={"ID":"b43cd4be-8675-4420-af28-f91e8e0d04b5","Type":"ContainerStarted","Data":"7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"}
Oct 31 00:13:03.737185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.737107    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" event={"ID":"b43cd4be-8675-4420-af28-f91e8e0d04b5","Type":"ContainerStarted","Data":"ea7ba27ea4f73ddff42309ef67f2a2dc487690a13c0c6a56b1a7b395656b4470"}
Oct 31 00:13:03.755649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.755600    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podStartSLOduration=1.7555847820000001 podStartE2EDuration="1.755584782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:03.754731903 +0000 UTC m=+1151.701102621" watchObservedRunningTime="2026-10-31 00:13:03.755584782 +0000 UTC m=+1151.701955495"
Oct 31 00:13:03.779414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.779368    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podStartSLOduration=1.779354924 podStartE2EDuration="1.779354924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:03.778545933 +0000 UTC m=+1151.724916651" watchObservedRunningTime="2026-10-31 00:13:03.779354924 +0000 UTC m=+1151.725725641"
Oct 31 00:13:03.804548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.804500    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podStartSLOduration=1.8044857410000001 podStartE2EDuration="1.804485741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:03.803585204 +0000 UTC m=+1151.749955923" watchObservedRunningTime="2026-10-31 00:13:03.804485741 +0000 UTC m=+1151.750856457"
Oct 31 00:13:03.834454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.834390    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podStartSLOduration=1.834370661 podStartE2EDuration="1.834370661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:03.831130052 +0000 UTC m=+1151.777500770" watchObservedRunningTime="2026-10-31 00:13:03.834370661 +0000 UTC m=+1151.780741379"
Oct 31 00:13:03.964511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:03.964454    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:04.157443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.157406    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:04.162321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.162300    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:04.640033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.640004    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:04.645708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.645670    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:04.727809 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.727780    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:04.733334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.733309    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:04.743186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.743163    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:04.743385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.743213    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:04.743524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.743502    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:04.743645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.743575    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:04.744095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.744080    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:04.744150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.744125    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:04.966661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:04.966573    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:05.748433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:05.748407    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:05.749117 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:05.749100    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:06.393707 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:13:06.393662    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:13:08.460093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.460062    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"]
Oct 31 00:13:08.463768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.463750    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.478189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.478168    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-7c5da3\""
Oct 31 00:13:08.479554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.479533    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-7c5da3-dockercfg-sxrt5\""
Oct 31 00:13:08.480516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.480496    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"]
Oct 31 00:13:08.543003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.542972    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff258\" (UniqueName: \"kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258\") pod \"payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.543173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.543092    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.644598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.644562    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff258\" (UniqueName: \"kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258\") pod \"payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.644777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.644665    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.645381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.645354    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.658111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.658081    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff258\" (UniqueName: \"kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258\") pod \"payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:08.777575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:08.777501    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:09.041863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.041784    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"]
Oct 31 00:13:09.046740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.046719    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.055856 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.055827    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c60826\""
Oct 31 00:13:09.057832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.057816    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c60826-dockercfg-924cl\""
Oct 31 00:13:09.065708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.065682    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"]
Oct 31 00:13:09.076421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.076390    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"]
Oct 31 00:13:09.103555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.103518    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"]
Oct 31 00:13:09.109555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.109532    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.115098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.115077    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"]
Oct 31 00:13:09.148948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.148922    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5mng\" (UniqueName: \"kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng\") pod \"payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.149128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.149107    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.251006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.250963    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5mng\" (UniqueName: \"kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng\") pod \"payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.254013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.253984    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.255820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.255106    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.256071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.256051    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlkf\" (UniqueName: \"kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf\") pod \"payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.263662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.255009    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.273062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.273030    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5mng\" (UniqueName: \"kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng\") pod \"payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.358774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.358681    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.358774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.358730    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlkf\" (UniqueName: \"kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf\") pod \"payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.359437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.359413    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.363433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.363408    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.374914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.374882    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlkf\" (UniqueName: \"kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf\") pod \"payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.432747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.432718    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:09.658628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.658592    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"]
Oct 31 00:13:09.665888 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:09.665854    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651895ba_2216_4409_9a0a_3c190aed5f52.slice/crio-4b31f15220027b52fa6e092bca014cb967f570c194c000674327b3af329a352c WatchSource:0}: Error finding container 4b31f15220027b52fa6e092bca014cb967f570c194c000674327b3af329a352c: Status 404 returned error can't find the container with id 4b31f15220027b52fa6e092bca014cb967f570c194c000674327b3af329a352c
Oct 31 00:13:09.745268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.745244    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"]
Oct 31 00:13:09.748284 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:09.748241    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34836a0f_c53e_4dee_a76b_85a954663e7a.slice/crio-3c039819dcce83d8fc3254ea24f0be5e759083c915528e909500a6b47660b055 WatchSource:0}: Error finding container 3c039819dcce83d8fc3254ea24f0be5e759083c915528e909500a6b47660b055: Status 404 returned error can't find the container with id 3c039819dcce83d8fc3254ea24f0be5e759083c915528e909500a6b47660b055
Oct 31 00:13:09.771813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.771788    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" event={"ID":"1ff2da4c-667d-4f1d-8994-0a2a972b43b3","Type":"ContainerStarted","Data":"b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837"}
Oct 31 00:13:09.771927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.771845    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" event={"ID":"1ff2da4c-667d-4f1d-8994-0a2a972b43b3","Type":"ContainerStarted","Data":"18bce8d07bfcd89ff2abae41bd3b2ae3ffbaa1cb5f3fcd7397a3f920c3e59dfe"}
Oct 31 00:13:09.771927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.771912    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:09.773279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.773249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" event={"ID":"34836a0f-c53e-4dee-a76b-85a954663e7a","Type":"ContainerStarted","Data":"3c039819dcce83d8fc3254ea24f0be5e759083c915528e909500a6b47660b055"}
Oct 31 00:13:09.775066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.775028    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" event={"ID":"651895ba-2216-4409-9a0a-3c190aed5f52","Type":"ContainerStarted","Data":"470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804"}
Oct 31 00:13:09.775181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.775072    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" event={"ID":"651895ba-2216-4409-9a0a-3c190aed5f52","Type":"ContainerStarted","Data":"4b31f15220027b52fa6e092bca014cb967f570c194c000674327b3af329a352c"}
Oct 31 00:13:09.775181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.775147    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:09.807948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.807895    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podStartSLOduration=1.807878632 podStartE2EDuration="1.807878632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:09.806527688 +0000 UTC m=+1157.752898415" watchObservedRunningTime="2026-10-31 00:13:09.807878632 +0000 UTC m=+1157.754249350"
Oct 31 00:13:09.832220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:09.832170    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podStartSLOduration=0.83215789 podStartE2EDuration="832.15789ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:09.829404778 +0000 UTC m=+1157.775775506" watchObservedRunningTime="2026-10-31 00:13:09.83215789 +0000 UTC m=+1157.778528607"
Oct 31 00:13:10.130509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.130420    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"]
Oct 31 00:13:10.134373 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.134347    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.151229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.151206    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d91de6\""
Oct 31 00:13:10.161866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.161836    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"]
Oct 31 00:13:10.272272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.272231    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.272495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.272435    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vc7p\" (UniqueName: \"kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p\") pod \"payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.375797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.375748    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vc7p\" (UniqueName: \"kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p\") pod \"payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.379717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.379690    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.381920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.380547    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.404225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.404188    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vc7p\" (UniqueName: \"kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p\") pod \"payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.448971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.448945    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:10.733113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.733087    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"]
Oct 31 00:13:10.735194 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:10.735170    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b84a98f_261e_4eb0_bdc3_ab5c60b200c4.slice/crio-857542eb30ec342a3234d379a0df1dd06e92e5b9e9fe2670d3d6e204616bb720 WatchSource:0}: Error finding container 857542eb30ec342a3234d379a0df1dd06e92e5b9e9fe2670d3d6e204616bb720: Status 404 returned error can't find the container with id 857542eb30ec342a3234d379a0df1dd06e92e5b9e9fe2670d3d6e204616bb720
Oct 31 00:13:10.794220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.794188    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" event={"ID":"34836a0f-c53e-4dee-a76b-85a954663e7a","Type":"ContainerStarted","Data":"3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d"}
Oct 31 00:13:10.794370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.794282    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:10.796428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.796350    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" event={"ID":"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4","Type":"ContainerStarted","Data":"857542eb30ec342a3234d379a0df1dd06e92e5b9e9fe2670d3d6e204616bb720"}
Oct 31 00:13:10.824640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.824590    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podStartSLOduration=1.824576012 podStartE2EDuration="1.824576012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:10.82209643 +0000 UTC m=+1158.768467149" watchObservedRunningTime="2026-10-31 00:13:10.824576012 +0000 UTC m=+1158.770946729"
Oct 31 00:13:10.909339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.909305    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"]
Oct 31 00:13:10.913382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.913361    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:10.919800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.919777    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-19adb9-dockercfg-vctbd\""
Oct 31 00:13:10.927553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.927528    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-19adb9\""
Oct 31 00:13:10.928163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.928144    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"]
Oct 31 00:13:10.934026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.934003    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"]
Oct 31 00:13:10.937748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.937732    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:10.948729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:10.948705    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"]
Oct 31 00:13:11.090366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.090267    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.090567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.090373    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zzl\" (UniqueName: \"kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl\") pod \"payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.090567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.090547    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.090703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.090586    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62lj\" (UniqueName: \"kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj\") pod \"payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.191703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.191666    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.191896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.191718    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62lj\" (UniqueName: \"kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj\") pod \"payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.191896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.191753    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.191896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.191805    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zzl\" (UniqueName: \"kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl\") pod \"payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.192557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.192522    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.194488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.194455    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.208348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.208321    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62lj\" (UniqueName: \"kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj\") pod \"payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.210986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.210962    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zzl\" (UniqueName: \"kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl\") pod \"payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.232093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.232069    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.253905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.253870    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.547942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.547908    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"]
Oct 31 00:13:11.553694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.553671    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.587604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.587578    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"]
Oct 31 00:13:11.587733 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:11.587623    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08725c5f_43e2_4fb4_8cb6_fd0549615d07.slice/crio-f2e3fff57c82d904c22b0b03bc8844d4c4e9ea60793955550a169b01e8e62d96 WatchSource:0}: Error finding container f2e3fff57c82d904c22b0b03bc8844d4c4e9ea60793955550a169b01e8e62d96: Status 404 returned error can't find the container with id f2e3fff57c82d904c22b0b03bc8844d4c4e9ea60793955550a169b01e8e62d96
Oct 31 00:13:11.590387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.590364    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"]
Oct 31 00:13:11.614791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.614762    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-0402df\""
Oct 31 00:13:11.616360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.616278    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-0402df-dockercfg-m2sz5\""
Oct 31 00:13:11.622331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.622047    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"]
Oct 31 00:13:11.672867 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:11.672829    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e43b897_6637_4e7d_8faa_0650c2d1cfef.slice/crio-3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616 WatchSource:0}: Error finding container 3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616: Status 404 returned error can't find the container with id 3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616
Oct 31 00:13:11.702977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.702684    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.702977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.702731    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254lg\" (UniqueName: \"kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg\") pod \"payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.804623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.804591    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.805109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.804643    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254lg\" (UniqueName: \"kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg\") pod \"payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.805498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.805437    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.806633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.806600    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" event={"ID":"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4","Type":"ContainerStarted","Data":"42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231"}
Oct 31 00:13:11.806794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.806769    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:11.808358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.808334    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" event={"ID":"1e43b897-6637-4e7d-8faa-0650c2d1cfef","Type":"ContainerStarted","Data":"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"}
Oct 31 00:13:11.808473 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.808379    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" event={"ID":"1e43b897-6637-4e7d-8faa-0650c2d1cfef","Type":"ContainerStarted","Data":"3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616"}
Oct 31 00:13:11.808473 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.808451    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:11.810145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.810124    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" event={"ID":"08725c5f-43e2-4fb4-8cb6-fd0549615d07","Type":"ContainerStarted","Data":"2b01d87a1755dcf35835c595c30550a6872ae58269423156f68be84f30801498"}
Oct 31 00:13:11.810253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.810166    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" event={"ID":"08725c5f-43e2-4fb4-8cb6-fd0549615d07","Type":"ContainerStarted","Data":"f2e3fff57c82d904c22b0b03bc8844d4c4e9ea60793955550a169b01e8e62d96"}
Oct 31 00:13:11.810343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.810323    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:11.838996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.838528    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254lg\" (UniqueName: \"kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg\") pod \"payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.866222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.866109    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podStartSLOduration=1.866088613 podStartE2EDuration="1.866088613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:11.865455257 +0000 UTC m=+1159.811825975" watchObservedRunningTime="2026-10-31 00:13:11.866088613 +0000 UTC m=+1159.812459331"
Oct 31 00:13:11.867997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.867723    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:11.877484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.869158    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podStartSLOduration=1.869146851 podStartE2EDuration="1.869146851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:11.83880131 +0000 UTC m=+1159.785172028" watchObservedRunningTime="2026-10-31 00:13:11.869146851 +0000 UTC m=+1159.815517569"
Oct 31 00:13:11.887495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:11.887415    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podStartSLOduration=1.887401467 podStartE2EDuration="1.887401467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:11.886515632 +0000 UTC m=+1159.832886373" watchObservedRunningTime="2026-10-31 00:13:11.887401467 +0000 UTC m=+1159.833772184"
Oct 31 00:13:12.172813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:12.172782    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"]
Oct 31 00:13:12.174331 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:12.174306    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72076e34_5810_4c87_ae9a_6bf9e3456b71.slice/crio-4794cd1c6e448cae8d11bcf1727455dee43a3df80228d0d2722e086dd68d5abe WatchSource:0}: Error finding container 4794cd1c6e448cae8d11bcf1727455dee43a3df80228d0d2722e086dd68d5abe: Status 404 returned error can't find the container with id 4794cd1c6e448cae8d11bcf1727455dee43a3df80228d0d2722e086dd68d5abe
Oct 31 00:13:12.822786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:12.822750    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" event={"ID":"72076e34-5810-4c87-ae9a-6bf9e3456b71","Type":"ContainerStarted","Data":"503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5"}
Oct 31 00:13:12.823929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:12.823900    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:12.824061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:12.823941    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" event={"ID":"72076e34-5810-4c87-ae9a-6bf9e3456b71","Type":"ContainerStarted","Data":"4794cd1c6e448cae8d11bcf1727455dee43a3df80228d0d2722e086dd68d5abe"}
Oct 31 00:13:12.846694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:12.846652    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podStartSLOduration=1.8466380610000002 podStartE2EDuration="1.846638061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:12.844355464 +0000 UTC m=+1160.790726181" watchObservedRunningTime="2026-10-31 00:13:12.846638061 +0000 UTC m=+1160.793008778"
Oct 31 00:13:17.917788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.917746    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"]
Oct 31 00:13:17.923872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.923848    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"]
Oct 31 00:13:17.924179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.924110    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:17.929793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.929768    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:13:17.930324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.930277    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:13:17.930781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.930756    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-4c360e-serving-cert\""
Oct 31 00:13:17.932163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.932111    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 00:13:17.932163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:17.932115    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ldwtt\""
Oct 31 00:13:18.021322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.021282    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.021551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.021435    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qbz\" (UniqueName: \"kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.021551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.021535    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.021700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.021678    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.122861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.122829    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.123026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.122899    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qbz\" (UniqueName: \"kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.123026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.122926    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.123159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.123086    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.123729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.123699    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.125893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.125872    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.125981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.125926    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.138428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.138402    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qbz\" (UniqueName: \"kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz\") pod \"maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.241622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.241543    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:18.563182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.563156    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"]
Oct 31 00:13:18.564239 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:18.564213    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314a02c0_fdfd_4053_a213_5b8691190a5a.slice/crio-a0608e0ec8ad7e1ec6d77be150f3c360970d8ed26e160b18d31a49de0e9be452 WatchSource:0}: Error finding container a0608e0ec8ad7e1ec6d77be150f3c360970d8ed26e160b18d31a49de0e9be452: Status 404 returned error can't find the container with id a0608e0ec8ad7e1ec6d77be150f3c360970d8ed26e160b18d31a49de0e9be452
Oct 31 00:13:18.858404 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:18.858321    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" event={"ID":"314a02c0-fdfd-4053-a213-5b8691190a5a","Type":"ContainerStarted","Data":"a0608e0ec8ad7e1ec6d77be150f3c360970d8ed26e160b18d31a49de0e9be452"}
Oct 31 00:13:20.804543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:20.804295    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:13:20.804543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:20.804441    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:13:21.844600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:21.842688    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:13:21.962614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:21.962143    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" event={"ID":"314a02c0-fdfd-4053-a213-5b8691190a5a","Type":"ContainerStarted","Data":"3cd8f54c987c481753e4108b677a271ac5b2d85caa54a56a77a467f6e569ecc9"}
Oct 31 00:13:21.962927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:21.962868    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:22.045608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:22.044212    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podStartSLOduration=2.098472037 podStartE2EDuration="5.044186741s" totalImagesPullingTime="2.945714704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:22.002973552 +0000 UTC m=+1169.949344271" watchObservedRunningTime="2026-10-31 00:13:22.044186741 +0000 UTC m=+1169.990557459"
Oct 31 00:13:22.829880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:22.829348    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:13:22.829880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:22.829498    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:13:22.830167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:22.829947    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:13:24.869147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:24.867950    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:13:28.755995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.754815    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"]
Oct 31 00:13:28.777564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.776435    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:28.777564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.776719    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"]
Oct 31 00:13:28.790354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.787030    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"kube-root-ca.crt\""
Oct 31 00:13:28.791212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.791192    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"default-dockercfg-vfp5x\""
Oct 31 00:13:28.791500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.791483    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"simulator-w5-7b2341-kserve-self-signed-certs\""
Oct 31 00:13:28.814382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.814321    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"openshift-service-ca.crt\""
Oct 31 00:13:28.911524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.910958    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:28.911524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.911020    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:28.911524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.911049    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:28.911524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.911186    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmjb\" (UniqueName: \"kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:28.911524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.911270    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:28.911524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:28.911327    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.015331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.015217    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.015331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.015296    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.015599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.015345    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.015599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.015374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.015599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.015557    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmjb\" (UniqueName: \"kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.018230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.015664    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.018230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.016058    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.018230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.017646    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.018420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.018273    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.020006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.019976    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.027635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.027604    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.069072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.063233    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmjb\" (UniqueName: \"kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb\") pod \"simulator-w5-7b2341-kserve-c87fb5b59-svr9n\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") " pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.099146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.097222    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:13:29.118220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.118181    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:29.468593 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:29.468560    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da4edd2_f1d2_478b_9bf1_1d5f3a3afa9f.slice/crio-9e6e8797ee9705e2786f658fe587a6212f5e800cb02ca5dc08e50e027f11d7b2 WatchSource:0}: Error finding container 9e6e8797ee9705e2786f658fe587a6212f5e800cb02ca5dc08e50e027f11d7b2: Status 404 returned error can't find the container with id 9e6e8797ee9705e2786f658fe587a6212f5e800cb02ca5dc08e50e027f11d7b2
Oct 31 00:13:29.468922 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:29.468893    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"]
Oct 31 00:13:30.133183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:30.133102    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" event={"ID":"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f","Type":"ContainerStarted","Data":"2ceb9a86170ccb669d35be64fbdd04adf0494932305afc3a62fda264aea95901"}
Oct 31 00:13:30.133183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:30.133154    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" event={"ID":"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f","Type":"ContainerStarted","Data":"9e6e8797ee9705e2786f658fe587a6212f5e800cb02ca5dc08e50e027f11d7b2"}
Oct 31 00:13:31.119107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:31.119065    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:33.153203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.153147    5078 generic.go:356] "Generic (PLEG): container finished" podID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerID="edad74b1e307270b0d17d11a44f9221935caeb1c1df5382db1f63c3fd0f37305" exitCode=0
Oct 31 00:13:33.153203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.153203    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" event={"ID":"cd56ff0f-da7a-4de9-b2a0-125e471de6d2","Type":"ContainerDied","Data":"edad74b1e307270b0d17d11a44f9221935caeb1c1df5382db1f63c3fd0f37305"}
Oct 31 00:13:33.153801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.153740    5078 scope.go:122] "RemoveContainer" containerID="edad74b1e307270b0d17d11a44f9221935caeb1c1df5382db1f63c3fd0f37305"
Oct 31 00:13:33.155294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.155268    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:33.155380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.155334    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:33.155380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.155364    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:33.218437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:33.217724    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podStartSLOduration=5.217704857 podStartE2EDuration="5.217704857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:30.158289065 +0000 UTC m=+1178.104659783" watchObservedRunningTime="2026-10-31 00:13:33.217704857 +0000 UTC m=+1181.164075575"
Oct 31 00:13:34.160741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.160704    5078 generic.go:356] "Generic (PLEG): container finished" podID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerID="558a54efa4466a8231c82bbe4ba518f3e330770b35d8ae092de99be0843a513b" exitCode=0
Oct 31 00:13:34.161257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.160762    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" event={"ID":"48402b00-cb93-46fe-87c6-4e1437aca83c","Type":"ContainerDied","Data":"558a54efa4466a8231c82bbe4ba518f3e330770b35d8ae092de99be0843a513b"}
Oct 31 00:13:34.163163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.163132    5078 generic.go:356] "Generic (PLEG): container finished" podID="f3aadd02-d322-476b-893b-2459b5f12808" containerID="f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755" exitCode=0
Oct 31 00:13:34.163287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.163271    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" event={"ID":"f3aadd02-d322-476b-893b-2459b5f12808","Type":"ContainerDied","Data":"f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"}
Oct 31 00:13:34.163962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.163946    5078 scope.go:122] "RemoveContainer" containerID="558a54efa4466a8231c82bbe4ba518f3e330770b35d8ae092de99be0843a513b"
Oct 31 00:13:34.164243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.164217    5078 scope.go:122] "RemoveContainer" containerID="f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"
Oct 31 00:13:34.165306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165275    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:34.165424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165362    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:34.165424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165405    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:34.165718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165694    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:34.165777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165756    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:34.165832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165788    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:34.165832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.165789    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" event={"ID":"cd56ff0f-da7a-4de9-b2a0-125e471de6d2","Type":"ContainerStarted","Data":"e64dda186588cf6928fa3f315c023d5e6bcd5c045bb5e83c21f38cda3401cfa8"}
Oct 31 00:13:34.647812 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.647778    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:34.648013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.647907    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:34.648013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.647948    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:34.749699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.749598    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:34.750032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.750015    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:34.750221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:34.750198    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:35.211514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.204503    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" event={"ID":"f3aadd02-d322-476b-893b-2459b5f12808","Type":"ContainerStarted","Data":"ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2"}
Oct 31 00:13:35.239619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.236166    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" event={"ID":"48402b00-cb93-46fe-87c6-4e1437aca83c","Type":"ContainerStarted","Data":"bede096fa117f7f58ead2dc017c904d6e94345f71ed05806efbe7f23f682ed6b"}
Oct 31 00:13:35.548751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.548662    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"]
Oct 31 00:13:35.553828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.553799    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.664534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.664020    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"openshift-service-ca.crt\""
Oct 31 00:13:35.685353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.685314    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:35.693731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.693694    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:35.695907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.687707    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"default-dockercfg-v7xzz\""
Oct 31 00:13:35.695907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.686299    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"]
Oct 31 00:13:35.695907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.688131    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"kube-root-ca.crt\""
Oct 31 00:13:35.736334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.732531    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"simulator-w4-d91de6-kserve-self-signed-certs\""
Oct 31 00:13:35.736334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.732690    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:35.815776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.815678    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.815776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.815745    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.816011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.815819    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.816011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.815875    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.816011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.815905    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.816162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.816057    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xxc\" (UniqueName: \"kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.816403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.816374    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:35.845490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.816444    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.985541    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.985593    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.985729    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xxc\" (UniqueName: \"kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.985831    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.985870    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.985940    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.986451    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:35.991489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.986799    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:36.003494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:35.996876    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:36.044495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.043428    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:36.045268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.045236    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:36.078628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.078548    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xxc\" (UniqueName: \"kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc\") pod \"simulator-w4-d91de6-kserve-5656898d8-pdcdk\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") " pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:36.162200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.162163    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:36.162630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.162548    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:36.162832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.162808    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:36.367540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.366686    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:36.674936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.671576    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:36.674936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.671640    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:36.710990 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:13:36.710950    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:13:36.735894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.735857    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:36.736182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:36.736149    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:37.541560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.448517    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"]
Oct 31 00:13:37.695489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.647449    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:37.695489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.684023    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:37.695489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.682661    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:37.695489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.684842    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:37.767533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.750816    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:37.767533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:37.750884    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:38.102590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.101022    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" event={"ID":"66848bbb-e0ec-48b2-9154-55eec47a4122","Type":"ContainerStarted","Data":"c2805bf59dd98e564946b21ce1c52bdbbefbe8497bff148f487a2a638e4964d5"}
Oct 31 00:13:38.179632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.169648    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:38.179632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.169711    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:38.644422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.644393    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:38.667499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.644974    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:38.740019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.739983    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:38.740275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.740248    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:38.824487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.823809    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"]
Oct 31 00:13:38.848487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.837981    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:38.861226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.857303    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"openshift-service-ca.crt\""
Oct 31 00:13:38.861226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.857435    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"default-dockercfg-h6gq5\""
Oct 31 00:13:38.904542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.864401    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"kube-root-ca.crt\""
Oct 31 00:13:38.904542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.864536    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"simulator-w2-7c5da3-kserve-self-signed-certs\""
Oct 31 00:13:38.920503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.918321    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:38.920503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:38.918385    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.108555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.073121    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"]
Oct 31 00:13:39.108555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.104189    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.108555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.104245    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.108555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.104315    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.108555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.104441    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27mk\" (UniqueName: \"kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.181545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.178192    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.181545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.178267    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.190511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.187738    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.191499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.191448    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:39.199513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.192240    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:39.199513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.193522    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:39.318504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.281637    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.318504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.281880    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27mk\" (UniqueName: \"kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.318504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.281986    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.318504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.282031    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.318504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.282911    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.318504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.284752    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.344487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.342749    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.412288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.412249    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27mk\" (UniqueName: \"kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.413039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.413013    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm\") pod \"simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") " pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.513814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.513709    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:39.647103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.646644    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:39.647103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.646715    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:39.681133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.680823    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:39.732014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.731573    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:39.732014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:39.731629    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:40.350874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.294811    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:40.350874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.294879    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:40.447757 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.416135    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"]
Oct 31 00:13:40.495788 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:40.495742    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641786fc_cd71_417f_a8b2_153079f8b512.slice/crio-acc50cb40b40f2629acc3da6542ae637cc3cded8bbbce21b7aadeeaa05875390 WatchSource:0}: Error finding container acc50cb40b40f2629acc3da6542ae637cc3cded8bbbce21b7aadeeaa05875390: Status 404 returned error can't find the container with id acc50cb40b40f2629acc3da6542ae637cc3cded8bbbce21b7aadeeaa05875390
Oct 31 00:13:40.543737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.543652    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" event={"ID":"66848bbb-e0ec-48b2-9154-55eec47a4122","Type":"ContainerStarted","Data":"47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42"}
Oct 31 00:13:40.564986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.564057    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:13:40.567613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.567545    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podStartSLOduration=5.56752676 podStartE2EDuration="5.56752676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:40.564843342 +0000 UTC m=+1188.511214062" watchObservedRunningTime="2026-10-31 00:13:40.56752676 +0000 UTC m=+1188.513897478"
Oct 31 00:13:40.644272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.643938    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:13:40.647953 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.647909    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:13:40.733839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.733805    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:40.734124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:40.734092    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:41.158374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.158339    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:41.158707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.158660    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:41.381547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.380551    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:41.548539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.548496    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" event={"ID":"641786fc-cd71-417f-a8b2-153079f8b512","Type":"ContainerStarted","Data":"76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb"}
Oct 31 00:13:41.548713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.548606    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" event={"ID":"641786fc-cd71-417f-a8b2-153079f8b512","Type":"ContainerStarted","Data":"acc50cb40b40f2629acc3da6542ae637cc3cded8bbbce21b7aadeeaa05875390"}
Oct 31 00:13:41.649154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.649123    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:41.651580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.651556    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:13:41.725455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.720341    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podStartSLOduration=3.72030876 podStartE2EDuration="3.72030876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:41.600686016 +0000 UTC m=+1189.547056727" watchObservedRunningTime="2026-10-31 00:13:41.72030876 +0000 UTC m=+1189.666679530"
Oct 31 00:13:41.729884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.729852    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:13:41.730105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:41.730079    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:13:42.157886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:42.157837    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:13:42.158090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:42.157927    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:13:42.515727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:42.515624    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:43.162274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.162242    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:43.162719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.162673    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:43.163441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.163422    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:13:43.273392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.273349    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:43.274439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.274410    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:43.279010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.278944    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:13:43.374051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.374006    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"]
Oct 31 00:13:43.378593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.378566    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.389171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.389075    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"kube-root-ca.crt\""
Oct 31 00:13:43.392823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.392784    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"openshift-service-ca.crt\""
Oct 31 00:13:43.393351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.393226    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"default-dockercfg-lpfcw\""
Oct 31 00:13:43.393351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.393318    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"simulator-w6-0402df-kserve-self-signed-certs\""
Oct 31 00:13:43.398560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.398513    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"]
Oct 31 00:13:43.484572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.484488    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.484721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.484571    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828gb\" (UniqueName: \"kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.484721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.484688    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.484795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.484774    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.484945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.484928    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.485149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.485001    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591120    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591210    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591281    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591380    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828gb\" (UniqueName: \"kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591415    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591479    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.591892    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.592397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.592090    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.594536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.594199    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.597641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.597590    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.602336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.602288    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.609326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.609296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828gb\" (UniqueName: \"kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb\") pod \"simulator-w6-0402df-kserve-84cd889645-8tqm4\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") " pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:43.698941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.698906    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:44.000173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:43.995050    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"]
Oct 31 00:13:44.000173 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:43.997013    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0dc259b_ccb1_4912_9dc8_c52852f55e75.slice/crio-9e00c984f3fcc884cf0761f409cc629e3b622b0d80128b5cd9e8d0468ef01124 WatchSource:0}: Error finding container 9e00c984f3fcc884cf0761f409cc629e3b622b0d80128b5cd9e8d0468ef01124: Status 404 returned error can't find the container with id 9e00c984f3fcc884cf0761f409cc629e3b622b0d80128b5cd9e8d0468ef01124
Oct 31 00:13:44.607541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:44.607508    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" event={"ID":"b0dc259b-ccb1-4912-9dc8-c52852f55e75","Type":"ContainerStarted","Data":"43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71"}
Oct 31 00:13:44.608015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:44.607558    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" event={"ID":"b0dc259b-ccb1-4912-9dc8-c52852f55e75","Type":"ContainerStarted","Data":"9e00c984f3fcc884cf0761f409cc629e3b622b0d80128b5cd9e8d0468ef01124"}
Oct 31 00:13:44.630164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:44.630117    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podStartSLOduration=1.6301043100000001 podStartE2EDuration="1.63010431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:44.62996226 +0000 UTC m=+1192.576332977" watchObservedRunningTime="2026-10-31 00:13:44.63010431 +0000 UTC m=+1192.576475026"
Oct 31 00:13:44.699251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:44.699223    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:45.616827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:45.616743    5078 generic.go:356] "Generic (PLEG): container finished" podID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerID="7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a" exitCode=0
Oct 31 00:13:45.617255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:45.616823    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" event={"ID":"b43cd4be-8675-4420-af28-f91e8e0d04b5","Type":"ContainerDied","Data":"7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"}
Oct 31 00:13:45.617316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:45.617294    5078 scope.go:122] "RemoveContainer" containerID="7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"
Oct 31 00:13:45.618843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:45.618813    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:45.618941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:45.618882    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:45.618941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:45.618917    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:13:46.368522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.368489    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:46.416822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.416786    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:46.635337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.634880    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" event={"ID":"b43cd4be-8675-4420-af28-f91e8e0d04b5","Type":"ContainerStarted","Data":"c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0"}
Oct 31 00:13:46.653371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.653337    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:13:46.965553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.963505    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:46.965553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.963647    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:13:46.965553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:46.963694    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:13:47.961887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:47.961851    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:13:47.962292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:47.961911    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:13:48.963165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:48.962099    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:13:48.963165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:48.962194    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:13:49.519485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:49.518987    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:49.540210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:49.539500    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:49.710837 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:49.710805    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:13:49.962531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:49.962497    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:13:49.962760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:49.962573    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:13:50.748573 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:50.748541    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:50.961608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:50.961565    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:13:50.961814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:50.961640    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:13:51.966251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:51.966224    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:51.967444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:51.967425    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:13:52.315770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:52.315683    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:13:52.332445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:52.332418    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:13:53.699344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:53.699306    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:53.712080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:53.712057    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:54.700648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:54.700617    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:13:57.350345 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.350312    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"]
Oct 31 00:13:57.356268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.356244    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.367793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.367761    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"premium-w5-7b2341-kserve-self-signed-certs\""
Oct 31 00:13:57.369776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.369518    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"]
Oct 31 00:13:57.409568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.409534    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.409742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.409607    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.409742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.409727    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.409832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.409808    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9pm\" (UniqueName: \"kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.409881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.409845    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.409932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.409918    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.511903    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512001    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9pm\" (UniqueName: \"kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512124    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512507    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512534    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.512637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512551    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.513298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512775    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.513298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.512833    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.515946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.515924    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.517759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.517735    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.526213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.526191    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9pm\" (UniqueName: \"kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm\") pod \"premium-w5-7b2341-kserve-bb7676bf4-2xzq4\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") " pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.671035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.671002    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:13:57.960933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:57.960852    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"]
Oct 31 00:13:57.964918 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:13:57.964890    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70a9ea32_a1a7_41eb_b335_1203dc85faaa.slice/crio-f86c846875cc7d87ea36ad5dd7357e035f000e4e883f74baf6468bf84ad693b2 WatchSource:0}: Error finding container f86c846875cc7d87ea36ad5dd7357e035f000e4e883f74baf6468bf84ad693b2: Status 404 returned error can't find the container with id f86c846875cc7d87ea36ad5dd7357e035f000e4e883f74baf6468bf84ad693b2
Oct 31 00:13:58.724691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:58.724656    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" event={"ID":"70a9ea32-a1a7-41eb-b335-1203dc85faaa","Type":"ContainerStarted","Data":"19f4559f3fe68403e292c5c93282060e525effc83e79d616d4672e16929314d7"}
Oct 31 00:13:58.725093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:58.724700    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" event={"ID":"70a9ea32-a1a7-41eb-b335-1203dc85faaa","Type":"ContainerStarted","Data":"f86c846875cc7d87ea36ad5dd7357e035f000e4e883f74baf6468bf84ad693b2"}
Oct 31 00:13:58.751048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:58.750989    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podStartSLOduration=1.7509689210000001 podStartE2EDuration="1.750968921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:58.745649107 +0000 UTC m=+1206.692019825" watchObservedRunningTime="2026-10-31 00:13:58.750968921 +0000 UTC m=+1206.697339639"
Oct 31 00:13:59.672181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:13:59.672150    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:14:06.387107 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:14:06.387067    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:14:07.672091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:07.672059    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:14:07.685862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:07.685837    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:14:07.797059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:07.797028    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:14:13.234331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.234298    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"]
Oct 31 00:14:13.239593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.239571    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.249250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.249223    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"premium-w4-d91de6-kserve-self-signed-certs\""
Oct 31 00:14:13.251620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.251594    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"]
Oct 31 00:14:13.411485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.411389    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.411698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.411599    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.411698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.411662    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.411827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.411805    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.411881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.411858    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd86d\" (UniqueName: \"kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.413271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.411976    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.513659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.513568    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.513659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.513637    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd86d\" (UniqueName: \"kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.513893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.513698    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.513893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.513731    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.514613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.514576    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.517598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.517571    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.525800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.525765    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.526267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.526245    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.526349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.526312    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.526729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.526710    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.531499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.531442    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.532211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.532181    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd86d\" (UniqueName: \"kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d\") pod \"premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") " pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.556361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.556340    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:13.845002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:13.844974    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"]
Oct 31 00:14:13.845424 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:14:13.845398    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5c46e27_4680_4481_9471_a06b9965262a.slice/crio-f9eb1dc9d62a419d25986ff042c7aec28cde06cb2f7617d59a233c9e523d48a1 WatchSource:0}: Error finding container f9eb1dc9d62a419d25986ff042c7aec28cde06cb2f7617d59a233c9e523d48a1: Status 404 returned error can't find the container with id f9eb1dc9d62a419d25986ff042c7aec28cde06cb2f7617d59a233c9e523d48a1
Oct 31 00:14:14.832409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:14.832376    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" event={"ID":"f5c46e27-4680-4481-9471-a06b9965262a","Type":"ContainerStarted","Data":"7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469"}
Oct 31 00:14:14.832931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:14.832421    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" event={"ID":"f5c46e27-4680-4481-9471-a06b9965262a","Type":"ContainerStarted","Data":"f9eb1dc9d62a419d25986ff042c7aec28cde06cb2f7617d59a233c9e523d48a1"}
Oct 31 00:14:14.855825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:14.855781    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podStartSLOduration=1.8557670910000001 podStartE2EDuration="1.855767091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:14.853959578 +0000 UTC m=+1222.800330296" watchObservedRunningTime="2026-10-31 00:14:14.855767091 +0000 UTC m=+1222.802137808"
Oct 31 00:14:15.557444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:15.557412    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:23.556853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.556814    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:23.572277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.572252    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:23.805139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.805105    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"]
Oct 31 00:14:23.810803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.810747    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.817453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.817428    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"]
Oct 31 00:14:23.821641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.821616    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"openshift-service-ca.crt\""
Oct 31 00:14:23.821901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.821879    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"premium-w1-4c360e-kserve-self-signed-certs\""
Oct 31 00:14:23.821981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.821971    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"kube-root-ca.crt\""
Oct 31 00:14:23.822024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.821721    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"default-dockercfg-hkvmb\""
Oct 31 00:14:23.853477 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.853429    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82fh\" (UniqueName: \"kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.853611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.853565    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.853681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.853610    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.853741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.853683    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.853824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.853809    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.853883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.853829    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.921000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.920971    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:14:23.959486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.959421    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82fh\" (UniqueName: \"kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.959712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.959535    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.959712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.959575    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.959712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.959665    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.959895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.959827    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.959895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.959851    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.960329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.960299    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.961109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.961082    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.964645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.961634    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.972593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.972558    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.987928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.987903    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:23.991540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:23.991415    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82fh\" (UniqueName: \"kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh\") pod \"premium-w1-4c360e-kserve-74fd457656-4pqtd\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") " pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:24.127256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:24.127168    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:24.468002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:24.467000    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"]
Oct 31 00:14:24.468002 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:14:24.467275    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75cf6aa4_c426_47d8_a8b2_fe5d95c1e22f.slice/crio-31d62ec50e4bd1048fb583f11c5797c051252b7b5eeb1f1c15284c4f7cc2da5d WatchSource:0}: Error finding container 31d62ec50e4bd1048fb583f11c5797c051252b7b5eeb1f1c15284c4f7cc2da5d: Status 404 returned error can't find the container with id 31d62ec50e4bd1048fb583f11c5797c051252b7b5eeb1f1c15284c4f7cc2da5d
Oct 31 00:14:24.916140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:24.916104    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" event={"ID":"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f","Type":"ContainerStarted","Data":"e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7"}
Oct 31 00:14:24.916680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:24.916649    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" event={"ID":"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f","Type":"ContainerStarted","Data":"31d62ec50e4bd1048fb583f11c5797c051252b7b5eeb1f1c15284c4f7cc2da5d"}
Oct 31 00:14:24.944683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:24.944627    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podStartSLOduration=1.9446103799999999 podStartE2EDuration="1.94461038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:24.941656806 +0000 UTC m=+1232.888027529" watchObservedRunningTime="2026-10-31 00:14:24.94461038 +0000 UTC m=+1232.890981103"
Oct 31 00:14:25.127627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:25.127598    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:34.127293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:34.127260    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:34.140115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:34.140094    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:34.997499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:34.997448    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:14:36.390953 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:14:36.390913    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:14:47.698486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.697510    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"]
Oct 31 00:14:47.703786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.703751    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.716051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.716030    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4c360e-openshift-default-dockercfg-22rlx\""
Oct 31 00:14:47.724265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.724231    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"]
Oct 31 00:14:47.839079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839038    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839099    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839280    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839348    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839433    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839489    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839521    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjm4\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839607    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.839701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.839670    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.933623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.933589    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"]
Oct 31 00:14:47.938093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.938061    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:47.940543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940520    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940559    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940612    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940644    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940693    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940710    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940734    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjm4\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940794    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.940937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940837    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.941204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.940942    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.941262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.941221    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.941718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.941333    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.941718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.941398    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.942332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.942305    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.943788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.943767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.943933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.943909    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.951580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.951524    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0402df-openshift-default-dockercfg-q78hj\""
Oct 31 00:14:47.960889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.959990    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"]
Oct 31 00:14:47.963614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.963588    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjm4\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:47.967587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:47.967561    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:48.041752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.041715    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:48.042359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042326    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75lk\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042387    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042424    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042444    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042606    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042641    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042656    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042672    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.042733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.042690    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145074    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75lk\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145192    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145272    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145325    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145644    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145726    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145808    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.145858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.145841    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.146358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.146329    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.147443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.147401    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.147789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.147764    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.147877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.147839    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.148042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.148019    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.149055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.149012    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.150691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.150669    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.159919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.159886    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75lk\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.160839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.160790    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.275271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.275176    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:48.339692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.337057    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"]
Oct 31 00:14:48.344149 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:14:48.344117    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b20778e_6adf_46f7_ae50_b389aa95f36c.slice/crio-bf4fa4bced80e756bbdf2237523a96f869d2b7ba9278bb4c35a829dc9ec2372d WatchSource:0}: Error finding container bf4fa4bced80e756bbdf2237523a96f869d2b7ba9278bb4c35a829dc9ec2372d: Status 404 returned error can't find the container with id bf4fa4bced80e756bbdf2237523a96f869d2b7ba9278bb4c35a829dc9ec2372d
Oct 31 00:14:48.354497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.352695    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:48.354497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.352783    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:48.354497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.352834    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:48.579412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.579320    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"]
Oct 31 00:14:48.591725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.589662    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:48.591725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.589747    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:48.591725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:48.589794    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:49.140159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.140120    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" event={"ID":"5de1af60-a588-4e7e-b8cc-b201871af022","Type":"ContainerStarted","Data":"f6312f987189427af8c5d0b6ff23e16589b9415380b519078678948536318780"}
Oct 31 00:14:49.140698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.140212    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" event={"ID":"5de1af60-a588-4e7e-b8cc-b201871af022","Type":"ContainerStarted","Data":"e1ddf97877753977b92607fd5775ae5a4fa96f38e534caa3d9b27309872a50c6"}
Oct 31 00:14:49.142375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.142344    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" event={"ID":"4b20778e-6adf-46f7-ae50-b389aa95f36c","Type":"ContainerStarted","Data":"7574b855fe378adf5c0b684ff4677fe8da41102cd5f14c0bd7d10319a59914da"}
Oct 31 00:14:49.142560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.142403    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" event={"ID":"4b20778e-6adf-46f7-ae50-b389aa95f36c","Type":"ContainerStarted","Data":"bf4fa4bced80e756bbdf2237523a96f869d2b7ba9278bb4c35a829dc9ec2372d"}
Oct 31 00:14:49.179449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.179382    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podStartSLOduration=2.179363786 podStartE2EDuration="2.179363786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:49.179265172 +0000 UTC m=+1257.125635890" watchObservedRunningTime="2026-10-31 00:14:49.179363786 +0000 UTC m=+1257.125734505"
Oct 31 00:14:49.230570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.230497    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podStartSLOduration=2.230477949 podStartE2EDuration="2.230477949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:49.226245973 +0000 UTC m=+1257.172616721" watchObservedRunningTime="2026-10-31 00:14:49.230477949 +0000 UTC m=+1257.176848672"
Oct 31 00:14:49.293920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.293887    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:49.295493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.294229    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:49.295740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:49.295709    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:50.043046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:50.043003    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:50.046839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:50.046807    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:14:50.047008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:50.046861    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:14:50.294544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:50.287451    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:50.294544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:50.287534    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:51.042820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:51.042771    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:14:51.043042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:51.042903    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:14:51.277485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:51.276822    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:51.277485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:51.276878    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:52.043321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:52.042987    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:14:52.043321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:52.043036    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:14:52.288665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:52.287968    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:52.288665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:52.288026    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:53.043197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:53.043158    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:14:53.043524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:53.043455    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:14:53.276940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:53.276901    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:53.277235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:53.277203    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:54.042069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:54.042037    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:14:54.042289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:54.042092    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:14:54.305544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:54.298572    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:54.305544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:54.298651    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:55.042761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:55.042721    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:14:55.042949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:55.042775    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:14:55.276762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:55.276727    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:14:55.276974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:55.276950    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 00:14:56.047452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:56.047417    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:56.047936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:56.047862    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:56.048754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:56.048731    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:14:56.281060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:56.281029    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:56.281514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:56.281492    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:56.282317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:56.282290    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:14:59.128876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.128843    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"]
Oct 31 00:14:59.134253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.134234    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.139788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.139766    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:14:59.139788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.139784    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:14:59.141426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.141396    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"]
Oct 31 00:14:59.204819 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.204776    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.204982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.204862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.204982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.204917    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrkf\" (UniqueName: \"kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.306562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.306528    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.306736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.306572    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.306736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.306620    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrkf\" (UniqueName: \"kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.307363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.307328    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.309254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.309220    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.320358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.320339    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrkf\" (UniqueName: \"kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf\") pod \"maas-api-revoke-sub-00837c1d-28stb\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.450515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.450455    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:14:59.729855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.729824    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"]
Oct 31 00:14:59.730291 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:14:59.730266    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9eb8e1_aa8f_4469_a451_d79190c375c3.slice/crio-7bcb7e43b9baa3e169f710b25e76aad5c101f127c5dd42551eeeddb05d6fcff9 WatchSource:0}: Error finding container 7bcb7e43b9baa3e169f710b25e76aad5c101f127c5dd42551eeeddb05d6fcff9: Status 404 returned error can't find the container with id 7bcb7e43b9baa3e169f710b25e76aad5c101f127c5dd42551eeeddb05d6fcff9
Oct 31 00:14:59.732178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:14:59.732161    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:15:00.159771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.159742    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"]
Oct 31 00:15:00.164926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.164906    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.217705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.217670    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb9n\" (UniqueName: \"kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.218212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.218171    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.228608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.228585    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"]
Oct 31 00:15:00.239667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.239643    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"]
Oct 31 00:15:00.239976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.239964    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"]
Oct 31 00:15:00.240186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.239899    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.259234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.256570    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"]
Oct 31 00:15:00.259234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.256685    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.294488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.290265    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"]
Oct 31 00:15:00.303276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.303245    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb" event={"ID":"7d9eb8e1-aa8f-4469-a451-d79190c375c3","Type":"ContainerStarted","Data":"7bcb7e43b9baa3e169f710b25e76aad5c101f127c5dd42551eeeddb05d6fcff9"}
Oct 31 00:15:00.319897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.319871    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb9n\" (UniqueName: \"kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.320060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.319971    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.320060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.320009    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ps2h\" (UniqueName: \"kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.320184 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.320091    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-h2jx5\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.320245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.320196    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.320333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.320312    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmgs\" (UniqueName: \"kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs\") pod \"maas-api-key-cleanup-29890095-h2jx5\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.321296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.321263    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.336264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.336234    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb9n\" (UniqueName: \"kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.422026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.421943    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmgs\" (UniqueName: \"kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs\") pod \"maas-api-key-cleanup-29890095-h2jx5\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.422180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.422089    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.422180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.422120    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ps2h\" (UniqueName: \"kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.422180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.422162    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-h2jx5\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.422812 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.422786    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.422923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.422810    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-h2jx5\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.437932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.437894    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ps2h\" (UniqueName: \"kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.438090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.438067    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmgs\" (UniqueName: \"kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs\") pod \"maas-api-key-cleanup-29890095-h2jx5\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.479691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.479650    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:00.560214 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.560178    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:00.573607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.572577    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:00.771706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.771674    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"]
Oct 31 00:15:00.775592 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:00.775566    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8125fdfd_c65d_4762_8262_8a3dc1d63fd6.slice/crio-d3ded8689bfd2d3301640d0fe1a15ffa0fdb761329d850ed6ad8aa5ffecd3af5 WatchSource:0}: Error finding container d3ded8689bfd2d3301640d0fe1a15ffa0fdb761329d850ed6ad8aa5ffecd3af5: Status 404 returned error can't find the container with id d3ded8689bfd2d3301640d0fe1a15ffa0fdb761329d850ed6ad8aa5ffecd3af5
Oct 31 00:15:00.892505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.891343    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"]
Oct 31 00:15:00.895152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:00.895124    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"]
Oct 31 00:15:00.900665 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:00.900640    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7453a4_8978_4ace_ba43_1527b491e00c.slice/crio-7cfd6592d2a5672db9a2f4ff76ea9ccf830a2303344c23decccfb78f3ab93f27 WatchSource:0}: Error finding container 7cfd6592d2a5672db9a2f4ff76ea9ccf830a2303344c23decccfb78f3ab93f27: Status 404 returned error can't find the container with id 7cfd6592d2a5672db9a2f4ff76ea9ccf830a2303344c23decccfb78f3ab93f27
Oct 31 00:15:00.901676 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:00.901642    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode433e6d3_9d7d_4faa_854e_dc901323fe93.slice/crio-039ec6ec7bc22a5f4ab438a7929801e969191eaf061b99f0124fbf32dfe5eaae WatchSource:0}: Error finding container 039ec6ec7bc22a5f4ab438a7929801e969191eaf061b99f0124fbf32dfe5eaae: Status 404 returned error can't find the container with id 039ec6ec7bc22a5f4ab438a7929801e969191eaf061b99f0124fbf32dfe5eaae
Oct 31 00:15:01.312082 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:01.312037    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerStarted","Data":"d3ded8689bfd2d3301640d0fe1a15ffa0fdb761329d850ed6ad8aa5ffecd3af5"}
Oct 31 00:15:01.313929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:01.313900    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" event={"ID":"ea7453a4-8978-4ace-ba43-1527b491e00c","Type":"ContainerStarted","Data":"7cfd6592d2a5672db9a2f4ff76ea9ccf830a2303344c23decccfb78f3ab93f27"}
Oct 31 00:15:01.315605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:01.315577    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerStarted","Data":"039ec6ec7bc22a5f4ab438a7929801e969191eaf061b99f0124fbf32dfe5eaae"}
Oct 31 00:15:05.352442 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:05.352340    5078 generic.go:356] "Generic (PLEG): container finished" podID="7d9eb8e1-aa8f-4469-a451-d79190c375c3" containerID="93430db61b1b058db139d6fd089c3896f4a1c11bbc786766de9f15f76740a0b5" exitCode=0
Oct 31 00:15:05.352442 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:05.352426    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb" event={"ID":"7d9eb8e1-aa8f-4469-a451-d79190c375c3","Type":"ContainerDied","Data":"93430db61b1b058db139d6fd089c3896f4a1c11bbc786766de9f15f76740a0b5"}
Oct 31 00:15:05.354344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:05.354303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerStarted","Data":"ed61c4a6e35d8868fee4b5fcbf29c45327cca8e954cd1526c759efe73ceb0b9b"}
Oct 31 00:15:05.356060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:05.356037    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerStarted","Data":"1d8c9e7b32632583d478652902cf076659de9e3eada4b04d7903ad08508aa0de"}
Oct 31 00:15:05.400589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:05.400528    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" podStartSLOduration=1.822767184 podStartE2EDuration="5.400507002s" totalImagesPullingTime="3.577739818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:05.395997347 +0000 UTC m=+1273.342368065" watchObservedRunningTime="2026-10-31 00:15:05.400507002 +0000 UTC m=+1273.346877726"
Oct 31 00:15:06.374945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.374900    5078 generic.go:356] "Generic (PLEG): container finished" podID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerID="ed61c4a6e35d8868fee4b5fcbf29c45327cca8e954cd1526c759efe73ceb0b9b" exitCode=60
Oct 31 00:15:06.375532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.375093    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerDied","Data":"ed61c4a6e35d8868fee4b5fcbf29c45327cca8e954cd1526c759efe73ceb0b9b"}
Oct 31 00:15:06.375819 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.375780    5078 scope.go:122] "RemoveContainer" containerID="ed61c4a6e35d8868fee4b5fcbf29c45327cca8e954cd1526c759efe73ceb0b9b"
Oct 31 00:15:06.379226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.378910    5078 generic.go:356] "Generic (PLEG): container finished" podID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerID="1d8c9e7b32632583d478652902cf076659de9e3eada4b04d7903ad08508aa0de" exitCode=60
Oct 31 00:15:06.379226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.379008    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerDied","Data":"1d8c9e7b32632583d478652902cf076659de9e3eada4b04d7903ad08508aa0de"}
Oct 31 00:15:06.379519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.379361    5078 scope.go:122] "RemoveContainer" containerID="1d8c9e7b32632583d478652902cf076659de9e3eada4b04d7903ad08508aa0de"
Oct 31 00:15:06.383392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.383367    5078 generic.go:356] "Generic (PLEG): container finished" podID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerID="0015d805e43920ae037750e407d41845947c585df993c3a380d587b4450d6eed" exitCode=60
Oct 31 00:15:06.383998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.383980    5078 scope.go:122] "RemoveContainer" containerID="0015d805e43920ae037750e407d41845947c585df993c3a380d587b4450d6eed"
Oct 31 00:15:06.384291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.384272    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" event={"ID":"ea7453a4-8978-4ace-ba43-1527b491e00c","Type":"ContainerDied","Data":"0015d805e43920ae037750e407d41845947c585df993c3a380d587b4450d6eed"}
Oct 31 00:15:06.402687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.402631    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" podStartSLOduration=2.411786266 podStartE2EDuration="6.402612238s" totalImagesPullingTime="3.990825972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:05.419705028 +0000 UTC m=+1273.366075746" watchObservedRunningTime="2026-10-31 00:15:06.402612238 +0000 UTC m=+1274.348982956"
Oct 31 00:15:06.432084 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:06.431777    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:15:06.619790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.619760    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:15:06.725947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.725852    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca\") pod \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") "
Oct 31 00:15:06.726138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.726086    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf\" (UniqueName: \"kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf\") pod \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") "
Oct 31 00:15:06.726234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.726220    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca\") pod \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\" (UID: \"7d9eb8e1-aa8f-4469-a451-d79190c375c3\") "
Oct 31 00:15:06.726317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.726293    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca" pod "7d9eb8e1-aa8f-4469-a451-d79190c375c3" (UID: "7d9eb8e1-aa8f-4469-a451-d79190c375c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:06.726718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.726696    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:06.770356 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.770326    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca" pod "7d9eb8e1-aa8f-4469-a451-d79190c375c3" (UID: "7d9eb8e1-aa8f-4469-a451-d79190c375c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:06.770558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.770387    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf" pod "7d9eb8e1-aa8f-4469-a451-d79190c375c3" (UID: "7d9eb8e1-aa8f-4469-a451-d79190c375c3"). InnerVolumeSpecName "kube-api-access-dbrkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:06.828045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.828013    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrkf\" (UniqueName: \"kubernetes.io/projected/7d9eb8e1-aa8f-4469-a451-d79190c375c3-kube-api-access-dbrkf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:06.828045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:06.828041    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9eb8e1-aa8f-4469-a451-d79190c375c3-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:07.398876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.398834    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerStarted","Data":"b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e"}
Oct 31 00:15:07.401261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.401233    5078 generic.go:356] "Generic (PLEG): container finished" podID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerID="8a44ca1ad161fe8a73005c5f29685ec85ede3e86dde8a7591cedc6400b9d7eef" exitCode=60
Oct 31 00:15:07.401404 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.401316    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" event={"ID":"ea7453a4-8978-4ace-ba43-1527b491e00c","Type":"ContainerDied","Data":"8a44ca1ad161fe8a73005c5f29685ec85ede3e86dde8a7591cedc6400b9d7eef"}
Oct 31 00:15:07.401404 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.401385    5078 scope.go:122] "RemoveContainer" containerID="0015d805e43920ae037750e407d41845947c585df993c3a380d587b4450d6eed"
Oct 31 00:15:07.401654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.401633    5078 scope.go:122] "RemoveContainer" containerID="8a44ca1ad161fe8a73005c5f29685ec85ede3e86dde8a7591cedc6400b9d7eef"
Oct 31 00:15:07.402009 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:07.401969    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-h2jx5_odh-ai-gateway-infra(ea7453a4-8978-4ace-ba43-1527b491e00c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c"
Oct 31 00:15:07.402955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.402939    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"
Oct 31 00:15:07.402955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.402943    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb" event={"ID":"7d9eb8e1-aa8f-4469-a451-d79190c375c3","Type":"ContainerDied","Data":"7bcb7e43b9baa3e169f710b25e76aad5c101f127c5dd42551eeeddb05d6fcff9"}
Oct 31 00:15:07.403101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.402992    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bcb7e43b9baa3e169f710b25e76aad5c101f127c5dd42551eeeddb05d6fcff9"
Oct 31 00:15:07.405750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.405725    5078 generic.go:356] "Generic (PLEG): container finished" podID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerID="e343a60578ffe6ff2595671c1bf9931e3434f10655ee86c99df23171f2db28df" exitCode=60
Oct 31 00:15:07.405750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.405737    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerDied","Data":"e343a60578ffe6ff2595671c1bf9931e3434f10655ee86c99df23171f2db28df"}
Oct 31 00:15:07.406127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.406103    5078 scope.go:122] "RemoveContainer" containerID="e343a60578ffe6ff2595671c1bf9931e3434f10655ee86c99df23171f2db28df"
Oct 31 00:15:07.406365 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:07.406341    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s_odh-ai-gateway-infra(e433e6d3-9d7d-4faa-854e-dc901323fe93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93"
Oct 31 00:15:07.543954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:07.543931    5078 scope.go:122] "RemoveContainer" containerID="ed61c4a6e35d8868fee4b5fcbf29c45327cca8e954cd1526c759efe73ceb0b9b"
Oct 31 00:15:08.416297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:08.416256    5078 generic.go:356] "Generic (PLEG): container finished" podID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerID="b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e" exitCode=60
Oct 31 00:15:08.416787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:08.416340    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerDied","Data":"b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e"}
Oct 31 00:15:08.416787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:08.416405    5078 scope.go:122] "RemoveContainer" containerID="1d8c9e7b32632583d478652902cf076659de9e3eada4b04d7903ad08508aa0de"
Oct 31 00:15:08.416787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:08.416657    5078 scope.go:122] "RemoveContainer" containerID="b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e"
Oct 31 00:15:08.416985 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:08.416964    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp_odh-ai-gateway-infra(8125fdfd-c65d-4762-8262-8a3dc1d63fd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6"
Oct 31 00:15:08.419531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:08.419510    5078 scope.go:122] "RemoveContainer" containerID="8a44ca1ad161fe8a73005c5f29685ec85ede3e86dde8a7591cedc6400b9d7eef"
Oct 31 00:15:08.419884 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:08.419861    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-h2jx5_odh-ai-gateway-infra(ea7453a4-8978-4ace-ba43-1527b491e00c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c"
Oct 31 00:15:08.421541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:08.421520    5078 scope.go:122] "RemoveContainer" containerID="e343a60578ffe6ff2595671c1bf9931e3434f10655ee86c99df23171f2db28df"
Oct 31 00:15:08.421807 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:08.421789    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s_odh-ai-gateway-infra(e433e6d3-9d7d-4faa-854e-dc901323fe93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93"
Oct 31 00:15:09.430395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:09.430365    5078 scope.go:122] "RemoveContainer" containerID="b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e"
Oct 31 00:15:09.430844 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:09.430696    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp_odh-ai-gateway-infra(8125fdfd-c65d-4762-8262-8a3dc1d63fd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6"
Oct 31 00:15:14.357144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.351745    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"]
Oct 31 00:15:14.357144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.355195    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7d9eb8e1-aa8f-4469-a451-d79190c375c3" containerName="revoke-keys"
Oct 31 00:15:14.357144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.355220    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9eb8e1-aa8f-4469-a451-d79190c375c3" containerName="revoke-keys"
Oct 31 00:15:14.357144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.355358    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9eb8e1-aa8f-4469-a451-d79190c375c3" containerName="revoke-keys"
Oct 31 00:15:14.363570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.361858    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.363570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.363336    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"]
Oct 31 00:15:14.373410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.373330    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:14.393240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.393207    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll99k\" (UniqueName: \"kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.393439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.393417    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.393691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.393674    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.496073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.496039    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll99k\" (UniqueName: \"kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.496353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.496324    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.496633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.496600    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.497164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.497136    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.508845 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.507943    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.510567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.510544    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll99k\" (UniqueName: \"kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k\") pod \"maas-api-revoke-sub-c1c7fba5-cprd4\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:14.698409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:14.698382    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:15.021656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.021546    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"]
Oct 31 00:15:15.376612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.376582    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"]
Oct 31 00:15:15.383038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.383016    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.390054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.390032    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"]
Oct 31 00:15:15.420912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.420877    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9tf\" (UniqueName: \"kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.421099 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.420935    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.421170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.421113    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.504432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.504395    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4" event={"ID":"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e","Type":"ContainerStarted","Data":"170f2139feb331cbf4fa7d4f2e7bcbd6e837b47baf6f410124b007a08fb9e9d3"}
Oct 31 00:15:15.522966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.522936    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9tf\" (UniqueName: \"kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.523131 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.523003    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.523131 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.523087    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.523969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.523814    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.525531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.525510    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.542162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.542064    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9tf\" (UniqueName: \"kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf\") pod \"maas-api-revoke-sub-8f6bd73d-qdsfv\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:15.699114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:15.699070    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:16.011536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:16.011503    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"]
Oct 31 00:15:16.516640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:16.516606    5078 generic.go:356] "Generic (PLEG): container finished" podID="bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" containerID="097eb5ca201caafabdbd9293102aa425761ef28ee5db223722c07bea9e4b8fbd" exitCode=0
Oct 31 00:15:16.517125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:16.516790    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4" event={"ID":"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e","Type":"ContainerDied","Data":"097eb5ca201caafabdbd9293102aa425761ef28ee5db223722c07bea9e4b8fbd"}
Oct 31 00:15:16.519904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:16.519879    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv" event={"ID":"5c860a92-9fef-4d58-9a7a-74ccda427223","Type":"ContainerStarted","Data":"c65ec3e175a483513a7c89c9f78727cc473ead4f0980d5cb2a721743bdf731d8"}
Oct 31 00:15:16.520027 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:16.519936    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv" event={"ID":"5c860a92-9fef-4d58-9a7a-74ccda427223","Type":"ContainerStarted","Data":"03aa038972df80b223b3de41d0febe98cb294652d1ed897f6e0e74d24703d7fc"}
Oct 31 00:15:16.570488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:16.569697    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv" podStartSLOduration=1.569678212 podStartE2EDuration="1.569678212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:16.569091477 +0000 UTC m=+1284.515462195" watchObservedRunningTime="2026-10-31 00:15:16.569678212 +0000 UTC m=+1284.516048932"
Oct 31 00:15:17.527411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.527374    5078 generic.go:356] "Generic (PLEG): container finished" podID="5c860a92-9fef-4d58-9a7a-74ccda427223" containerID="c65ec3e175a483513a7c89c9f78727cc473ead4f0980d5cb2a721743bdf731d8" exitCode=0
Oct 31 00:15:17.527841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.527478    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv" event={"ID":"5c860a92-9fef-4d58-9a7a-74ccda427223","Type":"ContainerDied","Data":"c65ec3e175a483513a7c89c9f78727cc473ead4f0980d5cb2a721743bdf731d8"}
Oct 31 00:15:17.747902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.747874    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:17.871566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.867912    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca\") pod \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") "
Oct 31 00:15:17.871566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.868078    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k\" (UniqueName: \"kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k\") pod \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") "
Oct 31 00:15:17.871566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.868187    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca\") pod \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\" (UID: \"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e\") "
Oct 31 00:15:17.892557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.892516    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca" pod "bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" (UID: "bfcb0fee-1ff2-4f0f-aef5-882bc22a012e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:17.893196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.893169    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k" pod "bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" (UID: "bfcb0fee-1ff2-4f0f-aef5-882bc22a012e"). InnerVolumeSpecName "kube-api-access-ll99k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:17.896008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.895682    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca" pod "bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" (UID: "bfcb0fee-1ff2-4f0f-aef5-882bc22a012e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:17.969988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.969953    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:17.969988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.969983    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll99k\" (UniqueName: \"kubernetes.io/projected/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-kube-api-access-ll99k\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:17.970173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:17.969999    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.187712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.187682    5078 scope.go:122] "RemoveContainer" containerID="e343a60578ffe6ff2595671c1bf9931e3434f10655ee86c99df23171f2db28df"
Oct 31 00:15:18.535798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.535770    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"
Oct 31 00:15:18.536235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.535801    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4" event={"ID":"bfcb0fee-1ff2-4f0f-aef5-882bc22a012e","Type":"ContainerDied","Data":"170f2139feb331cbf4fa7d4f2e7bcbd6e837b47baf6f410124b007a08fb9e9d3"}
Oct 31 00:15:18.536235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.535849    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170f2139feb331cbf4fa7d4f2e7bcbd6e837b47baf6f410124b007a08fb9e9d3"
Oct 31 00:15:18.538080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.538057    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerStarted","Data":"d400dc657598ef1f37dc56ee01ba03a563ea9c0a9b582278b3f87e50905ff984"}
Oct 31 00:15:18.724341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.722594    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"]
Oct 31 00:15:18.724341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.723252    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" containerName="revoke-keys"
Oct 31 00:15:18.724341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.723266    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" containerName="revoke-keys"
Oct 31 00:15:18.724341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.723479    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" containerName="revoke-keys"
Oct 31 00:15:18.729342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.729317    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"]
Oct 31 00:15:18.729522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.729500    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.733252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.733227    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.781971    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf\" (UniqueName: \"kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf\") pod \"5c860a92-9fef-4d58-9a7a-74ccda427223\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") "
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.782235    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca\") pod \"5c860a92-9fef-4d58-9a7a-74ccda427223\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") "
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.782274    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca\") pod \"5c860a92-9fef-4d58-9a7a-74ccda427223\" (UID: \"5c860a92-9fef-4d58-9a7a-74ccda427223\") "
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.782385    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ctfc\" (UniqueName: \"kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.782499    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.782591    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.783505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.783294    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca" pod "5c860a92-9fef-4d58-9a7a-74ccda427223" (UID: "5c860a92-9fef-4d58-9a7a-74ccda427223"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:18.788986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.788877    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca" pod "5c860a92-9fef-4d58-9a7a-74ccda427223" (UID: "5c860a92-9fef-4d58-9a7a-74ccda427223"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:18.791597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.791570    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf" pod "5c860a92-9fef-4d58-9a7a-74ccda427223" (UID: "5c860a92-9fef-4d58-9a7a-74ccda427223"). InnerVolumeSpecName "kube-api-access-7n9tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:18.884233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884206    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ctfc\" (UniqueName: \"kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.884408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884327    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.884516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884443    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.884614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884600    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.884675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884624    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c860a92-9fef-4d58-9a7a-74ccda427223-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.884675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884642    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n9tf\" (UniqueName: \"kubernetes.io/projected/5c860a92-9fef-4d58-9a7a-74ccda427223-kube-api-access-7n9tf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.884981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.884950    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.886950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.886924    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:18.898154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:18.898135    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ctfc\" (UniqueName: \"kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc\") pod \"maas-api-revoke-sub-3a546585-br47z\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:19.045376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.045298    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:19.188026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.187766    5078 scope.go:122] "RemoveContainer" containerID="b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e"
Oct 31 00:15:19.189225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.188717    5078 scope.go:122] "RemoveContainer" containerID="8a44ca1ad161fe8a73005c5f29685ec85ede3e86dde8a7591cedc6400b9d7eef"
Oct 31 00:15:19.333935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.333909    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"]
Oct 31 00:15:19.334392 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:19.334366    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1baafdba_7832_4ece_a361_0c8dd68e54ae.slice/crio-4b027291c214c00ab624a9e33afcf7ccbf2a278e6a646d8c67cfe6e2e87fc1cd WatchSource:0}: Error finding container 4b027291c214c00ab624a9e33afcf7ccbf2a278e6a646d8c67cfe6e2e87fc1cd: Status 404 returned error can't find the container with id 4b027291c214c00ab624a9e33afcf7ccbf2a278e6a646d8c67cfe6e2e87fc1cd
Oct 31 00:15:19.546375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.546342    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z" event={"ID":"1baafdba-7832-4ece-a361-0c8dd68e54ae","Type":"ContainerStarted","Data":"4b027291c214c00ab624a9e33afcf7ccbf2a278e6a646d8c67cfe6e2e87fc1cd"}
Oct 31 00:15:19.548816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.548790    5078 generic.go:356] "Generic (PLEG): container finished" podID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerID="d400dc657598ef1f37dc56ee01ba03a563ea9c0a9b582278b3f87e50905ff984" exitCode=60
Oct 31 00:15:19.548959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.548842    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerDied","Data":"d400dc657598ef1f37dc56ee01ba03a563ea9c0a9b582278b3f87e50905ff984"}
Oct 31 00:15:19.548959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.548917    5078 scope.go:122] "RemoveContainer" containerID="e343a60578ffe6ff2595671c1bf9931e3434f10655ee86c99df23171f2db28df"
Oct 31 00:15:19.549380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.549364    5078 scope.go:122] "RemoveContainer" containerID="d400dc657598ef1f37dc56ee01ba03a563ea9c0a9b582278b3f87e50905ff984"
Oct 31 00:15:19.549738 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:19.549710    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s_odh-ai-gateway-infra(e433e6d3-9d7d-4faa-854e-dc901323fe93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93"
Oct 31 00:15:19.550512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.550452    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv" event={"ID":"5c860a92-9fef-4d58-9a7a-74ccda427223","Type":"ContainerDied","Data":"03aa038972df80b223b3de41d0febe98cb294652d1ed897f6e0e74d24703d7fc"}
Oct 31 00:15:19.550587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.550510    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"
Oct 31 00:15:19.550587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.550530    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03aa038972df80b223b3de41d0febe98cb294652d1ed897f6e0e74d24703d7fc"
Oct 31 00:15:19.578480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:19.574681    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"]
Oct 31 00:15:20.559764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.559678    5078 generic.go:356] "Generic (PLEG): container finished" podID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerID="60c7e8a28cb5ebe9ace3fe2f481b291e80addc85725e4c556fe30bbec5d732c7" exitCode=60
Oct 31 00:15:20.559764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.559749    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" event={"ID":"ea7453a4-8978-4ace-ba43-1527b491e00c","Type":"ContainerDied","Data":"60c7e8a28cb5ebe9ace3fe2f481b291e80addc85725e4c556fe30bbec5d732c7"}
Oct 31 00:15:20.560247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.559815    5078 scope.go:122] "RemoveContainer" containerID="8a44ca1ad161fe8a73005c5f29685ec85ede3e86dde8a7591cedc6400b9d7eef"
Oct 31 00:15:20.560247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.560202    5078 scope.go:122] "RemoveContainer" containerID="60c7e8a28cb5ebe9ace3fe2f481b291e80addc85725e4c556fe30bbec5d732c7"
Oct 31 00:15:20.560622 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:20.560602    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-h2jx5_odh-ai-gateway-infra(ea7453a4-8978-4ace-ba43-1527b491e00c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c"
Oct 31 00:15:20.563697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.563664    5078 generic.go:356] "Generic (PLEG): container finished" podID="1baafdba-7832-4ece-a361-0c8dd68e54ae" containerID="c405091f1fbb92f38b9e47343023b907646e1896fc7a12ce796cbe8ee2073dd7" exitCode=0
Oct 31 00:15:20.563789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.563771    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z" event={"ID":"1baafdba-7832-4ece-a361-0c8dd68e54ae","Type":"ContainerDied","Data":"c405091f1fbb92f38b9e47343023b907646e1896fc7a12ce796cbe8ee2073dd7"}
Oct 31 00:15:20.566290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.566252    5078 generic.go:356] "Generic (PLEG): container finished" podID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerID="fc9be7aac8e625a0eaeeaa564e36f2c3113d7678b22aaee10f41d7d86b33bcc5" exitCode=60
Oct 31 00:15:20.566400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.566308    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerDied","Data":"fc9be7aac8e625a0eaeeaa564e36f2c3113d7678b22aaee10f41d7d86b33bcc5"}
Oct 31 00:15:20.566721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.566706    5078 scope.go:122] "RemoveContainer" containerID="fc9be7aac8e625a0eaeeaa564e36f2c3113d7678b22aaee10f41d7d86b33bcc5"
Oct 31 00:15:20.567015 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:20.566987    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp_odh-ai-gateway-infra(8125fdfd-c65d-4762-8262-8a3dc1d63fd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6"
Oct 31 00:15:20.575021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.575002    5078 scope.go:122] "RemoveContainer" containerID="b4db266905423e21189a7f8e5fe124313b52b4fce45a4e718874e9a6a7d76f1e"
Oct 31 00:15:20.746580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.746542    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:20.823086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.822984    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca\") pod \"e433e6d3-9d7d-4faa-854e-dc901323fe93\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") "
Oct 31 00:15:20.823370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.823336    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h\" (UniqueName: \"kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h\") pod \"e433e6d3-9d7d-4faa-854e-dc901323fe93\" (UID: \"e433e6d3-9d7d-4faa-854e-dc901323fe93\") "
Oct 31 00:15:20.850773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.850728    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h" pod "e433e6d3-9d7d-4faa-854e-dc901323fe93" (UID: "e433e6d3-9d7d-4faa-854e-dc901323fe93"). InnerVolumeSpecName "kube-api-access-8ps2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:20.863144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.863111    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca" pod "e433e6d3-9d7d-4faa-854e-dc901323fe93" (UID: "e433e6d3-9d7d-4faa-854e-dc901323fe93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:20.924902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.924865    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ps2h\" (UniqueName: \"kubernetes.io/projected/e433e6d3-9d7d-4faa-854e-dc901323fe93-kube-api-access-8ps2h\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:20.924902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:20.924892    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e433e6d3-9d7d-4faa-854e-dc901323fe93-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:21.623492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.620646    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s" event={"ID":"e433e6d3-9d7d-4faa-854e-dc901323fe93","Type":"ContainerDied","Data":"039ec6ec7bc22a5f4ab438a7929801e969191eaf061b99f0124fbf32dfe5eaae"}
Oct 31 00:15:21.623492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.620723    5078 scope.go:122] "RemoveContainer" containerID="d400dc657598ef1f37dc56ee01ba03a563ea9c0a9b582278b3f87e50905ff984"
Oct 31 00:15:21.623492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.620819    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"
Oct 31 00:15:21.633073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.633046    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"]
Oct 31 00:15:21.677756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.677724    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"]
Oct 31 00:15:21.742488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.737439    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"]
Oct 31 00:15:21.742488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:21.737506    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890095-4mq7s"]
Oct 31 00:15:22.256781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.256753    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" path="/var/lib/kubelet/pods/e433e6d3-9d7d-4faa-854e-dc901323fe93/volumes"
Oct 31 00:15:22.389973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.389950    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:22.392162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.392135    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:22.394234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.394210    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:22.452038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.448620    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca\") pod \"1baafdba-7832-4ece-a361-0c8dd68e54ae\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") "
Oct 31 00:15:22.452038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.448997    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc\" (UniqueName: \"kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc\") pod \"1baafdba-7832-4ece-a361-0c8dd68e54ae\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") "
Oct 31 00:15:22.452038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.449711    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca" pod "1baafdba-7832-4ece-a361-0c8dd68e54ae" (UID: "1baafdba-7832-4ece-a361-0c8dd68e54ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:22.452038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.451995    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n\" (UniqueName: \"kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n\") pod \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") "
Oct 31 00:15:22.452328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.452058    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca\") pod \"ea7453a4-8978-4ace-ba43-1527b491e00c\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") "
Oct 31 00:15:22.452328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.452113    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca\") pod \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\" (UID: \"8125fdfd-c65d-4762-8262-8a3dc1d63fd6\") "
Oct 31 00:15:22.452328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.452142    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs\" (UniqueName: \"kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs\") pod \"ea7453a4-8978-4ace-ba43-1527b491e00c\" (UID: \"ea7453a4-8978-4ace-ba43-1527b491e00c\") "
Oct 31 00:15:22.452328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.452163    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca\") pod \"1baafdba-7832-4ece-a361-0c8dd68e54ae\" (UID: \"1baafdba-7832-4ece-a361-0c8dd68e54ae\") "
Oct 31 00:15:22.452870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.452741    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.452957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.452801    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca" pod "8125fdfd-c65d-4762-8262-8a3dc1d63fd6" (UID: "8125fdfd-c65d-4762-8262-8a3dc1d63fd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:22.453199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.453175    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc" pod "1baafdba-7832-4ece-a361-0c8dd68e54ae" (UID: "1baafdba-7832-4ece-a361-0c8dd68e54ae"). InnerVolumeSpecName "kube-api-access-2ctfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:22.453257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.453234    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca" pod "ea7453a4-8978-4ace-ba43-1527b491e00c" (UID: "ea7453a4-8978-4ace-ba43-1527b491e00c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:22.454658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.454636    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n" pod "8125fdfd-c65d-4762-8262-8a3dc1d63fd6" (UID: "8125fdfd-c65d-4762-8262-8a3dc1d63fd6"). InnerVolumeSpecName "kube-api-access-8mb9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:22.455437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.455413    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca" pod "1baafdba-7832-4ece-a361-0c8dd68e54ae" (UID: "1baafdba-7832-4ece-a361-0c8dd68e54ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:22.455804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.455785    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs" pod "ea7453a4-8978-4ace-ba43-1527b491e00c" (UID: "ea7453a4-8978-4ace-ba43-1527b491e00c"). InnerVolumeSpecName "kube-api-access-7hmgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:22.554058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.553971    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb9n\" (UniqueName: \"kubernetes.io/projected/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-kube-api-access-8mb9n\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.554058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.554010    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7453a4-8978-4ace-ba43-1527b491e00c-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.554058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.554026    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125fdfd-c65d-4762-8262-8a3dc1d63fd6-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.554058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.554039    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmgs\" (UniqueName: \"kubernetes.io/projected/ea7453a4-8978-4ace-ba43-1527b491e00c-kube-api-access-7hmgs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.554058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.554054    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1baafdba-7832-4ece-a361-0c8dd68e54ae-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.554443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.554067    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ctfc\" (UniqueName: \"kubernetes.io/projected/1baafdba-7832-4ece-a361-0c8dd68e54ae-kube-api-access-2ctfc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.669966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.662152    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z" event={"ID":"1baafdba-7832-4ece-a361-0c8dd68e54ae","Type":"ContainerDied","Data":"4b027291c214c00ab624a9e33afcf7ccbf2a278e6a646d8c67cfe6e2e87fc1cd"}
Oct 31 00:15:22.669966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.662200    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b027291c214c00ab624a9e33afcf7ccbf2a278e6a646d8c67cfe6e2e87fc1cd"
Oct 31 00:15:22.669966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.662265    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"
Oct 31 00:15:22.669966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.667330    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp" event={"ID":"8125fdfd-c65d-4762-8262-8a3dc1d63fd6","Type":"ContainerDied","Data":"d3ded8689bfd2d3301640d0fe1a15ffa0fdb761329d850ed6ad8aa5ffecd3af5"}
Oct 31 00:15:22.669966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.667387    5078 scope.go:122] "RemoveContainer" containerID="fc9be7aac8e625a0eaeeaa564e36f2c3113d7678b22aaee10f41d7d86b33bcc5"
Oct 31 00:15:22.669966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.667443    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"
Oct 31 00:15:22.672112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.672087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5" event={"ID":"ea7453a4-8978-4ace-ba43-1527b491e00c","Type":"ContainerDied","Data":"7cfd6592d2a5672db9a2f4ff76ea9ccf830a2303344c23decccfb78f3ab93f27"}
Oct 31 00:15:22.672204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.672165    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"
Oct 31 00:15:22.690930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.690911    5078 scope.go:122] "RemoveContainer" containerID="60c7e8a28cb5ebe9ace3fe2f481b291e80addc85725e4c556fe30bbec5d732c7"
Oct 31 00:15:22.724014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.723987    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"]
Oct 31 00:15:22.727474 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.727436    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-h2jx5"]
Oct 31 00:15:22.739739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.739716    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"]
Oct 31 00:15:22.744128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:22.744105    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890095-ndkgp"]
Oct 31 00:15:23.593160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593078    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"]
Oct 31 00:15:23.593887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593863    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.593887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593885    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593902    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593908    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593922    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593927    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593938    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1baafdba-7832-4ece-a361-0c8dd68e54ae" containerName="revoke-keys"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593942    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baafdba-7832-4ece-a361-0c8dd68e54ae" containerName="revoke-keys"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593954    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593958    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593968    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593971    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593981    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593985    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593991    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.593995    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594008    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5c860a92-9fef-4d58-9a7a-74ccda427223" containerName="revoke-keys"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594012    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c860a92-9fef-4d58-9a7a-74ccda427223" containerName="revoke-keys"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594108    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:23.594107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594116    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594125    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c860a92-9fef-4d58-9a7a-74ccda427223" containerName="revoke-keys"
Oct 31 00:15:23.594976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594134    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:23.594976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594139    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:23.594976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594146    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:23.594976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594151    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baafdba-7832-4ece-a361-0c8dd68e54ae" containerName="revoke-keys"
Oct 31 00:15:23.594976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.594159    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:23.599392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.599371    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.605223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.604989    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:23.606452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.606429    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"]
Oct 31 00:15:23.607122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.607103    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:15:23.674052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.674018    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.674516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.674164    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.674516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.674259    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnm2g\" (UniqueName: \"kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.776083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.776054    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.776315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.776125    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.776315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.776234    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnm2g\" (UniqueName: \"kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.776795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.776771    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.778762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.778740    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.789955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.789931    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnm2g\" (UniqueName: \"kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g\") pod \"maas-api-revoke-sub-a428e1fe-6q2vg\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:23.934376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:23.934337    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:24.195262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:24.195181    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" path="/var/lib/kubelet/pods/8125fdfd-c65d-4762-8262-8a3dc1d63fd6/volumes"
Oct 31 00:15:24.195737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:24.195721    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" path="/var/lib/kubelet/pods/ea7453a4-8978-4ace-ba43-1527b491e00c/volumes"
Oct 31 00:15:24.270735 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:24.270704    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0271e86b_904a_47f2_80fb_79e51d10a378.slice/crio-9043bdbdf8a1d39ec1dcef5ed5d532e8bc395d62388360a3f0c65fe7aa46a772 WatchSource:0}: Error finding container 9043bdbdf8a1d39ec1dcef5ed5d532e8bc395d62388360a3f0c65fe7aa46a772: Status 404 returned error can't find the container with id 9043bdbdf8a1d39ec1dcef5ed5d532e8bc395d62388360a3f0c65fe7aa46a772
Oct 31 00:15:24.270859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:24.270837    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"]
Oct 31 00:15:24.693178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:24.693151    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg" event={"ID":"0271e86b-904a-47f2-80fb-79e51d10a378","Type":"ContainerStarted","Data":"c801034770ddf9c68bf42feb3043a59fdac295ebfbf86c17071fffeb01cbe4fa"}
Oct 31 00:15:24.693636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:24.693205    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg" event={"ID":"0271e86b-904a-47f2-80fb-79e51d10a378","Type":"ContainerStarted","Data":"9043bdbdf8a1d39ec1dcef5ed5d532e8bc395d62388360a3f0c65fe7aa46a772"}
Oct 31 00:15:24.748407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:24.748303    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg" podStartSLOduration=1.748287886 podStartE2EDuration="1.748287886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:24.746400121 +0000 UTC m=+1292.692770838" watchObservedRunningTime="2026-10-31 00:15:24.748287886 +0000 UTC m=+1292.694658603"
Oct 31 00:15:25.703067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:25.703028    5078 generic.go:356] "Generic (PLEG): container finished" podID="0271e86b-904a-47f2-80fb-79e51d10a378" containerID="c801034770ddf9c68bf42feb3043a59fdac295ebfbf86c17071fffeb01cbe4fa" exitCode=0
Oct 31 00:15:25.703560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:25.703155    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg" event={"ID":"0271e86b-904a-47f2-80fb-79e51d10a378","Type":"ContainerDied","Data":"c801034770ddf9c68bf42feb3043a59fdac295ebfbf86c17071fffeb01cbe4fa"}
Oct 31 00:15:26.084233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084152    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"]
Oct 31 00:15:26.084769 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084755    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:26.084813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084771    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:26.084813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084797    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:26.084813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084801    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:26.084914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084899    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7453a4-8978-4ace-ba43-1527b491e00c" containerName="cleanup"
Oct 31 00:15:26.084914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084910    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e433e6d3-9d7d-4faa-854e-dc901323fe93" containerName="cleanup"
Oct 31 00:15:26.084975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.084920    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8125fdfd-c65d-4762-8262-8a3dc1d63fd6" containerName="cleanup"
Oct 31 00:15:26.088437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.088411    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.101202 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.101174    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"]
Oct 31 00:15:26.135961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.135924    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.136118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.136046    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.136231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.136215    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hjt\" (UniqueName: \"kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.237239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.237210    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hjt\" (UniqueName: \"kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.237403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.237279    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.237596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.237432    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.238495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.238450    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.240244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.240215    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.253746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.251615    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hjt\" (UniqueName: \"kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt\") pod \"maas-api-revoke-sub-90cd392b-l8cjb\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.275224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.275191    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"]
Oct 31 00:15:26.287969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.286975    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.297354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.295781    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"]
Oct 31 00:15:26.342352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.342280    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.342666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.342640    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.342885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.342866    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwfk\" (UniqueName: \"kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.406691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.406662    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:26.444621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.444589    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.444762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.444695    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwfk\" (UniqueName: \"kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.444893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.444877    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.445581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.445556    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.447196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.447167    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.461987 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.460931    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwfk\" (UniqueName: \"kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk\") pod \"maas-api-revoke-sub-3371ee39-wmpzc\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.629905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.629834    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:26.711144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.710953    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"]
Oct 31 00:15:26.912400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.912375    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:26.942150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.942120    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"]
Oct 31 00:15:26.942551 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:26.942524    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf946d93d_fd1c_49d7_af04_77c709bfdd7d.slice/crio-22ce2b14902a63a4c2af9f4e66d4856fea5ab3b7f31daef6179204dee1a69c5f WatchSource:0}: Error finding container 22ce2b14902a63a4c2af9f4e66d4856fea5ab3b7f31daef6179204dee1a69c5f: Status 404 returned error can't find the container with id 22ce2b14902a63a4c2af9f4e66d4856fea5ab3b7f31daef6179204dee1a69c5f
Oct 31 00:15:26.967367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.967339    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g\" (UniqueName: \"kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g\") pod \"0271e86b-904a-47f2-80fb-79e51d10a378\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") "
Oct 31 00:15:26.967561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.967542    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca\") pod \"0271e86b-904a-47f2-80fb-79e51d10a378\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") "
Oct 31 00:15:26.967632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.967612    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca\") pod \"0271e86b-904a-47f2-80fb-79e51d10a378\" (UID: \"0271e86b-904a-47f2-80fb-79e51d10a378\") "
Oct 31 00:15:26.967936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.967910    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca" pod "0271e86b-904a-47f2-80fb-79e51d10a378" (UID: "0271e86b-904a-47f2-80fb-79e51d10a378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:26.968146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.968129    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:26.969927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.969902    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca" pod "0271e86b-904a-47f2-80fb-79e51d10a378" (UID: "0271e86b-904a-47f2-80fb-79e51d10a378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:26.970479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:26.970434    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g" pod "0271e86b-904a-47f2-80fb-79e51d10a378" (UID: "0271e86b-904a-47f2-80fb-79e51d10a378"). InnerVolumeSpecName "kube-api-access-nnm2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:27.069921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.069887    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271e86b-904a-47f2-80fb-79e51d10a378-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.069921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.069922    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnm2g\" (UniqueName: \"kubernetes.io/projected/0271e86b-904a-47f2-80fb-79e51d10a378-kube-api-access-nnm2g\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.720739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.720710    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"
Oct 31 00:15:27.720739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.720732    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg" event={"ID":"0271e86b-904a-47f2-80fb-79e51d10a378","Type":"ContainerDied","Data":"9043bdbdf8a1d39ec1dcef5ed5d532e8bc395d62388360a3f0c65fe7aa46a772"}
Oct 31 00:15:27.721236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.720784    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9043bdbdf8a1d39ec1dcef5ed5d532e8bc395d62388360a3f0c65fe7aa46a772"
Oct 31 00:15:27.722646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.722613    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc" event={"ID":"f946d93d-fd1c-49d7-af04-77c709bfdd7d","Type":"ContainerStarted","Data":"e1416a54a84c615dccb9a0d1c36b533dda18aec2a6b9197118d2f2948b192d5f"}
Oct 31 00:15:27.722770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.722675    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc" event={"ID":"f946d93d-fd1c-49d7-af04-77c709bfdd7d","Type":"ContainerStarted","Data":"22ce2b14902a63a4c2af9f4e66d4856fea5ab3b7f31daef6179204dee1a69c5f"}
Oct 31 00:15:27.724217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.724194    5078 generic.go:356] "Generic (PLEG): container finished" podID="bb430dfd-014c-48cd-8d57-eab607347d3c" containerID="8acb12f34b39bd50886709b2e9c8425cc3552f916bb1efafcf09905cbe6827e6" exitCode=0
Oct 31 00:15:27.724323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.724280    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb" event={"ID":"bb430dfd-014c-48cd-8d57-eab607347d3c","Type":"ContainerDied","Data":"8acb12f34b39bd50886709b2e9c8425cc3552f916bb1efafcf09905cbe6827e6"}
Oct 31 00:15:27.724361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.724328    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb" event={"ID":"bb430dfd-014c-48cd-8d57-eab607347d3c","Type":"ContainerStarted","Data":"491025d724d5f7e7aa745e563fb53dc2de5493836c98f2edc8e9ab59e31d7bbf"}
Oct 31 00:15:27.750150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:27.750109    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc" podStartSLOduration=1.750097232 podStartE2EDuration="1.750097232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:27.747602225 +0000 UTC m=+1295.693972942" watchObservedRunningTime="2026-10-31 00:15:27.750097232 +0000 UTC m=+1295.696467949"
Oct 31 00:15:28.738297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:28.738260    5078 generic.go:356] "Generic (PLEG): container finished" podID="f946d93d-fd1c-49d7-af04-77c709bfdd7d" containerID="e1416a54a84c615dccb9a0d1c36b533dda18aec2a6b9197118d2f2948b192d5f" exitCode=0
Oct 31 00:15:28.738798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:28.738368    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc" event={"ID":"f946d93d-fd1c-49d7-af04-77c709bfdd7d","Type":"ContainerDied","Data":"e1416a54a84c615dccb9a0d1c36b533dda18aec2a6b9197118d2f2948b192d5f"}
Oct 31 00:15:28.978505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:28.977276    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:29.002253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.002173    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"]
Oct 31 00:15:29.002876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.002834    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0271e86b-904a-47f2-80fb-79e51d10a378" containerName="revoke-keys"
Oct 31 00:15:29.002876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.002854    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0271e86b-904a-47f2-80fb-79e51d10a378" containerName="revoke-keys"
Oct 31 00:15:29.003097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.002923    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bb430dfd-014c-48cd-8d57-eab607347d3c" containerName="revoke-keys"
Oct 31 00:15:29.003097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.002932    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb430dfd-014c-48cd-8d57-eab607347d3c" containerName="revoke-keys"
Oct 31 00:15:29.003201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.003111    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb430dfd-014c-48cd-8d57-eab607347d3c" containerName="revoke-keys"
Oct 31 00:15:29.003201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.003133    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0271e86b-904a-47f2-80fb-79e51d10a378" containerName="revoke-keys"
Oct 31 00:15:29.006594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.006574    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.026561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.026537    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"]
Oct 31 00:15:29.102960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.102927    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca\") pod \"bb430dfd-014c-48cd-8d57-eab607347d3c\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") "
Oct 31 00:15:29.103142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103129    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca\") pod \"bb430dfd-014c-48cd-8d57-eab607347d3c\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") "
Oct 31 00:15:29.103202 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103164    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt\" (UniqueName: \"kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt\") pod \"bb430dfd-014c-48cd-8d57-eab607347d3c\" (UID: \"bb430dfd-014c-48cd-8d57-eab607347d3c\") "
Oct 31 00:15:29.103514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103489    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.103691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103525    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca" pod "bb430dfd-014c-48cd-8d57-eab607347d3c" (UID: "bb430dfd-014c-48cd-8d57-eab607347d3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:29.103691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103615    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8sb\" (UniqueName: \"kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.103865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103690    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.104002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.103976    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:29.105760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.105737    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt" pod "bb430dfd-014c-48cd-8d57-eab607347d3c" (UID: "bb430dfd-014c-48cd-8d57-eab607347d3c"). InnerVolumeSpecName "kube-api-access-67hjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:29.105892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.105863    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca" pod "bb430dfd-014c-48cd-8d57-eab607347d3c" (UID: "bb430dfd-014c-48cd-8d57-eab607347d3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:29.177810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.177771    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"]
Oct 31 00:15:29.182224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.182198    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"]
Oct 31 00:15:29.182557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.182518    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.190746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.190603    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.197680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.197350    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"]
Oct 31 00:15:29.201747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.201609    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"]
Oct 31 00:15:29.205348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.205322    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.208048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.205390    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8sb\" (UniqueName: \"kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.208048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.205435    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.208048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.207355    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hjt\" (UniqueName: \"kubernetes.io/projected/bb430dfd-014c-48cd-8d57-eab607347d3c-kube-api-access-67hjt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:29.208048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.207379    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb430dfd-014c-48cd-8d57-eab607347d3c-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:29.209665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.208055    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.211699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.211416    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.223381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.223360    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8sb\" (UniqueName: \"kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb\") pod \"maas-api-revoke-sub-a043cfb7-cgvb2\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.308737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.308654    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.308872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.308801    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.308872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.308841    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5z7\" (UniqueName: \"kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.309035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.309012    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.309209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.309195    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.309263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.309250    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfptf\" (UniqueName: \"kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.334744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.334718    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:29.410818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.410786    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.410947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.410862    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.410947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.410901    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5z7\" (UniqueName: \"kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.411079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.410949    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.411079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.411042    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.411079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.411071    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfptf\" (UniqueName: \"kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.425086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.425055    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.427688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.427630    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfptf\" (UniqueName: \"kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.431313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.430613    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.433905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.433841    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.436794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.436732    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6accf90-psqmw\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.437037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.436984    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5z7\" (UniqueName: \"kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7\") pod \"maas-api-revoke-sub-a68b5313-vwwsg\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.515615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.515351    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:29.541977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.541944    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:29.671154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.668164    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"]
Oct 31 00:15:29.746326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.746304    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"
Oct 31 00:15:29.746880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.746342    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb" event={"ID":"bb430dfd-014c-48cd-8d57-eab607347d3c","Type":"ContainerDied","Data":"491025d724d5f7e7aa745e563fb53dc2de5493836c98f2edc8e9ab59e31d7bbf"}
Oct 31 00:15:29.746880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.746855    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491025d724d5f7e7aa745e563fb53dc2de5493836c98f2edc8e9ab59e31d7bbf"
Oct 31 00:15:29.747947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.747903    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2" event={"ID":"b7e25ea5-d81f-4b94-9a70-877a9759262d","Type":"ContainerStarted","Data":"69ae4dad56cad75124900c2347a29f7628d9dd8294f415b4045198e3a6b3f41a"}
Oct 31 00:15:29.808843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.808806    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"]
Oct 31 00:15:29.809212 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:29.809192    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd23fec6_99e3_4b7a_a8e8_645abe23ff50.slice/crio-adeefb68f9e4a70696a15f29576c6503dbe2d660b17ba0f4fc4862ef55710453 WatchSource:0}: Error finding container adeefb68f9e4a70696a15f29576c6503dbe2d660b17ba0f4fc4862ef55710453: Status 404 returned error can't find the container with id adeefb68f9e4a70696a15f29576c6503dbe2d660b17ba0f4fc4862ef55710453
Oct 31 00:15:29.856786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.856752    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"]
Oct 31 00:15:29.860598 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:29.860574    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a8d332_4698_4af0_b3c3_62b16b190adf.slice/crio-46425f7353975a164b350c6fbcd294b571280668686d0144fc8dba5248e877f0 WatchSource:0}: Error finding container 46425f7353975a164b350c6fbcd294b571280668686d0144fc8dba5248e877f0: Status 404 returned error can't find the container with id 46425f7353975a164b350c6fbcd294b571280668686d0144fc8dba5248e877f0
Oct 31 00:15:29.911427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:29.911405    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:30.025480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.025432    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk\" (UniqueName: \"kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk\") pod \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") "
Oct 31 00:15:30.025648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.025597    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca\") pod \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") "
Oct 31 00:15:30.025754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.025741    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca\") pod \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\" (UID: \"f946d93d-fd1c-49d7-af04-77c709bfdd7d\") "
Oct 31 00:15:30.025967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.025940    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca" pod "f946d93d-fd1c-49d7-af04-77c709bfdd7d" (UID: "f946d93d-fd1c-49d7-af04-77c709bfdd7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:30.026172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.026155    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:30.027971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.027942    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca" pod "f946d93d-fd1c-49d7-af04-77c709bfdd7d" (UID: "f946d93d-fd1c-49d7-af04-77c709bfdd7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:30.028081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.027968    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk" pod "f946d93d-fd1c-49d7-af04-77c709bfdd7d" (UID: "f946d93d-fd1c-49d7-af04-77c709bfdd7d"). InnerVolumeSpecName "kube-api-access-kqwfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:30.127532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.127499    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946d93d-fd1c-49d7-af04-77c709bfdd7d-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:30.127532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.127527    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwfk\" (UniqueName: \"kubernetes.io/projected/f946d93d-fd1c-49d7-af04-77c709bfdd7d-kube-api-access-kqwfk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:30.757668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.757623    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw" event={"ID":"bd23fec6-99e3-4b7a-a8e8-645abe23ff50","Type":"ContainerStarted","Data":"e340cd9a484d5f1a3e8ff858533d197d2a6422eb7f6b49bd94f30c7a974f6d48"}
Oct 31 00:15:30.758223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.757699    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw" event={"ID":"bd23fec6-99e3-4b7a-a8e8-645abe23ff50","Type":"ContainerStarted","Data":"adeefb68f9e4a70696a15f29576c6503dbe2d660b17ba0f4fc4862ef55710453"}
Oct 31 00:15:30.759422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.759393    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2" event={"ID":"b7e25ea5-d81f-4b94-9a70-877a9759262d","Type":"ContainerStarted","Data":"62e2d58217835bd6c7e1d61b7efc306bd0c133c4a94aac60b6c063da7d356da7"}
Oct 31 00:15:30.761018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.760997    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg" event={"ID":"55a8d332-4698-4af0-b3c3-62b16b190adf","Type":"ContainerStarted","Data":"6c98564594d3ef00f7b1df8bb6348e420f93af33766de7c20e652cc5bd1c3ad5"}
Oct 31 00:15:30.761121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.761039    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg" event={"ID":"55a8d332-4698-4af0-b3c3-62b16b190adf","Type":"ContainerStarted","Data":"46425f7353975a164b350c6fbcd294b571280668686d0144fc8dba5248e877f0"}
Oct 31 00:15:30.762622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.762599    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc" event={"ID":"f946d93d-fd1c-49d7-af04-77c709bfdd7d","Type":"ContainerDied","Data":"22ce2b14902a63a4c2af9f4e66d4856fea5ab3b7f31daef6179204dee1a69c5f"}
Oct 31 00:15:30.762622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.762613    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"
Oct 31 00:15:30.762747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.762644    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ce2b14902a63a4c2af9f4e66d4856fea5ab3b7f31daef6179204dee1a69c5f"
Oct 31 00:15:30.787803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.787740    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw" podStartSLOduration=1.7877212120000001 podStartE2EDuration="1.787721212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:30.782446974 +0000 UTC m=+1298.728817677" watchObservedRunningTime="2026-10-31 00:15:30.787721212 +0000 UTC m=+1298.734091929"
Oct 31 00:15:30.820899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.820831    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg" podStartSLOduration=1.8208119740000002 podStartE2EDuration="1.820811974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:30.818299551 +0000 UTC m=+1298.764670270" watchObservedRunningTime="2026-10-31 00:15:30.820811974 +0000 UTC m=+1298.767182692"
Oct 31 00:15:30.845647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:30.845586    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2" podStartSLOduration=2.84556752 podStartE2EDuration="2.84556752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:30.844492533 +0000 UTC m=+1298.790863241" watchObservedRunningTime="2026-10-31 00:15:30.84556752 +0000 UTC m=+1298.791938239"
Oct 31 00:15:31.793505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:31.792883    5078 generic.go:356] "Generic (PLEG): container finished" podID="bd23fec6-99e3-4b7a-a8e8-645abe23ff50" containerID="e340cd9a484d5f1a3e8ff858533d197d2a6422eb7f6b49bd94f30c7a974f6d48" exitCode=0
Oct 31 00:15:31.793505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:31.793000    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw" event={"ID":"bd23fec6-99e3-4b7a-a8e8-645abe23ff50","Type":"ContainerDied","Data":"e340cd9a484d5f1a3e8ff858533d197d2a6422eb7f6b49bd94f30c7a974f6d48"}
Oct 31 00:15:31.807489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:31.800766    5078 generic.go:356] "Generic (PLEG): container finished" podID="b7e25ea5-d81f-4b94-9a70-877a9759262d" containerID="62e2d58217835bd6c7e1d61b7efc306bd0c133c4a94aac60b6c063da7d356da7" exitCode=0
Oct 31 00:15:31.807489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:31.800928    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2" event={"ID":"b7e25ea5-d81f-4b94-9a70-877a9759262d","Type":"ContainerDied","Data":"62e2d58217835bd6c7e1d61b7efc306bd0c133c4a94aac60b6c063da7d356da7"}
Oct 31 00:15:31.831486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:31.815971    5078 generic.go:356] "Generic (PLEG): container finished" podID="55a8d332-4698-4af0-b3c3-62b16b190adf" containerID="6c98564594d3ef00f7b1df8bb6348e420f93af33766de7c20e652cc5bd1c3ad5" exitCode=0
Oct 31 00:15:31.831486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:31.816014    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg" event={"ID":"55a8d332-4698-4af0-b3c3-62b16b190adf","Type":"ContainerDied","Data":"6c98564594d3ef00f7b1df8bb6348e420f93af33766de7c20e652cc5bd1c3ad5"}
Oct 31 00:15:34.173640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.160067    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:34.173640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.162134    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2" event={"ID":"b7e25ea5-d81f-4b94-9a70-877a9759262d","Type":"ContainerDied","Data":"69ae4dad56cad75124900c2347a29f7628d9dd8294f415b4045198e3a6b3f41a"}
Oct 31 00:15:34.173640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.162216    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ae4dad56cad75124900c2347a29f7628d9dd8294f415b4045198e3a6b3f41a"
Oct 31 00:15:34.435488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.434587    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:34.454494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.437656    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:34.463941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.456219    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca\") pod \"b7e25ea5-d81f-4b94-9a70-877a9759262d\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") "
Oct 31 00:15:34.463941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.456265    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca\") pod \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") "
Oct 31 00:15:34.463941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.456307    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca\") pod \"55a8d332-4698-4af0-b3c3-62b16b190adf\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") "
Oct 31 00:15:34.463941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.457274    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca" pod "55a8d332-4698-4af0-b3c3-62b16b190adf" (UID: "55a8d332-4698-4af0-b3c3-62b16b190adf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:34.463941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.457408    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca" pod "b7e25ea5-d81f-4b94-9a70-877a9759262d" (UID: "b7e25ea5-d81f-4b94-9a70-877a9759262d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:34.463941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.459905    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca" pod "bd23fec6-99e3-4b7a-a8e8-645abe23ff50" (UID: "bd23fec6-99e3-4b7a-a8e8-645abe23ff50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:34.557890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.557854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7\" (UniqueName: \"kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7\") pod \"55a8d332-4698-4af0-b3c3-62b16b190adf\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") "
Oct 31 00:15:34.558120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.558103    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf\" (UniqueName: \"kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf\") pod \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") "
Oct 31 00:15:34.558205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.558153    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca\") pod \"55a8d332-4698-4af0-b3c3-62b16b190adf\" (UID: \"55a8d332-4698-4af0-b3c3-62b16b190adf\") "
Oct 31 00:15:34.558271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.558210    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca\") pod \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\" (UID: \"bd23fec6-99e3-4b7a-a8e8-645abe23ff50\") "
Oct 31 00:15:34.558634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.558350    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb\" (UniqueName: \"kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb\") pod \"b7e25ea5-d81f-4b94-9a70-877a9759262d\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") "
Oct 31 00:15:34.558634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.558491    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca\") pod \"b7e25ea5-d81f-4b94-9a70-877a9759262d\" (UID: \"b7e25ea5-d81f-4b94-9a70-877a9759262d\") "
Oct 31 00:15:34.559098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.559052    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.559098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.559084    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.559098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.559099    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.559694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.559656    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca" pod "bd23fec6-99e3-4b7a-a8e8-645abe23ff50" (UID: "bd23fec6-99e3-4b7a-a8e8-645abe23ff50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:34.561131 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.561097    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca" pod "55a8d332-4698-4af0-b3c3-62b16b190adf" (UID: "55a8d332-4698-4af0-b3c3-62b16b190adf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:34.561355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.561326    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf" pod "bd23fec6-99e3-4b7a-a8e8-645abe23ff50" (UID: "bd23fec6-99e3-4b7a-a8e8-645abe23ff50"). InnerVolumeSpecName "kube-api-access-qfptf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:34.561496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.561476    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca" pod "b7e25ea5-d81f-4b94-9a70-877a9759262d" (UID: "b7e25ea5-d81f-4b94-9a70-877a9759262d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:34.561644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.561622    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7" pod "55a8d332-4698-4af0-b3c3-62b16b190adf" (UID: "55a8d332-4698-4af0-b3c3-62b16b190adf"). InnerVolumeSpecName "kube-api-access-wc5z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:34.563223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.563203    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb" pod "b7e25ea5-d81f-4b94-9a70-877a9759262d" (UID: "b7e25ea5-d81f-4b94-9a70-877a9759262d"). InnerVolumeSpecName "kube-api-access-rn8sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:34.660223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.660182    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8sb\" (UniqueName: \"kubernetes.io/projected/b7e25ea5-d81f-4b94-9a70-877a9759262d-kube-api-access-rn8sb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.660223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.660225    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e25ea5-d81f-4b94-9a70-877a9759262d-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.660437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.660243    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5z7\" (UniqueName: \"kubernetes.io/projected/55a8d332-4698-4af0-b3c3-62b16b190adf-kube-api-access-wc5z7\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.660437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.660258    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfptf\" (UniqueName: \"kubernetes.io/projected/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-kube-api-access-qfptf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.660437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.660290    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a8d332-4698-4af0-b3c3-62b16b190adf-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:34.660437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:34.660305    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd23fec6-99e3-4b7a-a8e8-645abe23ff50-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:35.170933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.170901    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"
Oct 31 00:15:35.170933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.170919    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg" event={"ID":"55a8d332-4698-4af0-b3c3-62b16b190adf","Type":"ContainerDied","Data":"46425f7353975a164b350c6fbcd294b571280668686d0144fc8dba5248e877f0"}
Oct 31 00:15:35.171171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.170971    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46425f7353975a164b350c6fbcd294b571280668686d0144fc8dba5248e877f0"
Oct 31 00:15:35.172563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.172537    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"
Oct 31 00:15:35.172665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.172579    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw" event={"ID":"bd23fec6-99e3-4b7a-a8e8-645abe23ff50","Type":"ContainerDied","Data":"adeefb68f9e4a70696a15f29576c6503dbe2d660b17ba0f4fc4862ef55710453"}
Oct 31 00:15:35.172665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.172613    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adeefb68f9e4a70696a15f29576c6503dbe2d660b17ba0f4fc4862ef55710453"
Oct 31 00:15:35.172790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.172774    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"
Oct 31 00:15:35.964069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964038    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"]
Oct 31 00:15:35.964650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964636    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f946d93d-fd1c-49d7-af04-77c709bfdd7d" containerName="revoke-keys"
Oct 31 00:15:35.964712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964652    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f946d93d-fd1c-49d7-af04-77c709bfdd7d" containerName="revoke-keys"
Oct 31 00:15:35.964712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964668    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bd23fec6-99e3-4b7a-a8e8-645abe23ff50" containerName="revoke-keys"
Oct 31 00:15:35.964712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964673    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd23fec6-99e3-4b7a-a8e8-645abe23ff50" containerName="revoke-keys"
Oct 31 00:15:35.964712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964694    5078 cpu_manager.go:414] "containerMap: removing container" podUID="55a8d332-4698-4af0-b3c3-62b16b190adf" containerName="revoke-keys"
Oct 31 00:15:35.964712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964699    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a8d332-4698-4af0-b3c3-62b16b190adf" containerName="revoke-keys"
Oct 31 00:15:35.964853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964724    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b7e25ea5-d81f-4b94-9a70-877a9759262d" containerName="revoke-keys"
Oct 31 00:15:35.964853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964730    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e25ea5-d81f-4b94-9a70-877a9759262d" containerName="revoke-keys"
Oct 31 00:15:35.964853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964822    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a8d332-4698-4af0-b3c3-62b16b190adf" containerName="revoke-keys"
Oct 31 00:15:35.964853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964838    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e25ea5-d81f-4b94-9a70-877a9759262d" containerName="revoke-keys"
Oct 31 00:15:35.964853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964846    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f946d93d-fd1c-49d7-af04-77c709bfdd7d" containerName="revoke-keys"
Oct 31 00:15:35.964853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.964852    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd23fec6-99e3-4b7a-a8e8-645abe23ff50" containerName="revoke-keys"
Oct 31 00:15:35.969594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.969569    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:35.976135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.976048    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7cm\" (UniqueName: \"kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:35.977890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.977862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:35.978121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.976792    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"]
Oct 31 00:15:35.978713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.978683    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:35.979246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.979230    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:35.979420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:35.979408    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:15:36.080264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.080232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.080486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.080387    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.080570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.080498    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7cm\" (UniqueName: \"kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.081098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.081072    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.083358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.083328    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.101726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.101671    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7cm\" (UniqueName: \"kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm\") pod \"maas-api-revoke-sub-a945191b-p785v\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.295040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.294949    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:36.409961 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:15:36.409913    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:15:36.594121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:36.594097    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"]
Oct 31 00:15:36.597577 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:36.597544    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8073975_c499_4cb6_ba86_9db9de1356eb.slice/crio-7425c675c6f0cbe83b010acbc49aa287599743b1ffb675ab6adceec72a9aebe8 WatchSource:0}: Error finding container 7425c675c6f0cbe83b010acbc49aa287599743b1ffb675ab6adceec72a9aebe8: Status 404 returned error can't find the container with id 7425c675c6f0cbe83b010acbc49aa287599743b1ffb675ab6adceec72a9aebe8
Oct 31 00:15:37.190236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:37.190202    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v" event={"ID":"a8073975-c499-4cb6-ba86-9db9de1356eb","Type":"ContainerStarted","Data":"a186bbbe9dc4af77601d5f7ed33819a0ae4af69b8f6245f91218d49dd2066560"}
Oct 31 00:15:37.190873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:37.190258    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v" event={"ID":"a8073975-c499-4cb6-ba86-9db9de1356eb","Type":"ContainerStarted","Data":"7425c675c6f0cbe83b010acbc49aa287599743b1ffb675ab6adceec72a9aebe8"}
Oct 31 00:15:37.213500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:37.213434    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v" podStartSLOduration=2.213419838 podStartE2EDuration="2.213419838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:37.212492795 +0000 UTC m=+1305.158863503" watchObservedRunningTime="2026-10-31 00:15:37.213419838 +0000 UTC m=+1305.159790555"
Oct 31 00:15:38.199214 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.199183    5078 generic.go:356] "Generic (PLEG): container finished" podID="a8073975-c499-4cb6-ba86-9db9de1356eb" containerID="a186bbbe9dc4af77601d5f7ed33819a0ae4af69b8f6245f91218d49dd2066560" exitCode=0
Oct 31 00:15:38.199638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.199262    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v" event={"ID":"a8073975-c499-4cb6-ba86-9db9de1356eb","Type":"ContainerDied","Data":"a186bbbe9dc4af77601d5f7ed33819a0ae4af69b8f6245f91218d49dd2066560"}
Oct 31 00:15:38.759866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.759812    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"]
Oct 31 00:15:38.766059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.766034    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.770273 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.768517    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"]
Oct 31 00:15:38.820334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.820247    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.820602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.820572    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q28x\" (UniqueName: \"kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.820709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.820623    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.922304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.922270    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.922502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.922431    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q28x\" (UniqueName: \"kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.922502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.922481    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.923032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.922994    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.925167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.925146    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:38.936311 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:38.936282    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q28x\" (UniqueName: \"kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x\") pod \"maas-api-revoke-sub-c10c43b4-jnbwq\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:39.085822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.085737    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:39.388978 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:39.388930    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda168e155_83cf_4683_9e4e_a56681e0a24b.slice/crio-03933133da1b701aae96a819ab5734aa68c744a1e2694e08dde9c0d012919e3c WatchSource:0}: Error finding container 03933133da1b701aae96a819ab5734aa68c744a1e2694e08dde9c0d012919e3c: Status 404 returned error can't find the container with id 03933133da1b701aae96a819ab5734aa68c744a1e2694e08dde9c0d012919e3c
Oct 31 00:15:39.390582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.390557    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"]
Oct 31 00:15:39.446261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.446234    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:39.534272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.534236    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca\") pod \"a8073975-c499-4cb6-ba86-9db9de1356eb\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") "
Oct 31 00:15:39.534478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.534346    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca\") pod \"a8073975-c499-4cb6-ba86-9db9de1356eb\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") "
Oct 31 00:15:39.534478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.534395    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm\" (UniqueName: \"kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm\") pod \"a8073975-c499-4cb6-ba86-9db9de1356eb\" (UID: \"a8073975-c499-4cb6-ba86-9db9de1356eb\") "
Oct 31 00:15:39.534667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.534643    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca" pod "a8073975-c499-4cb6-ba86-9db9de1356eb" (UID: "a8073975-c499-4cb6-ba86-9db9de1356eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:39.534915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.534896    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:39.570888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.570839    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca" pod "a8073975-c499-4cb6-ba86-9db9de1356eb" (UID: "a8073975-c499-4cb6-ba86-9db9de1356eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:39.571383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.571356    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm" pod "a8073975-c499-4cb6-ba86-9db9de1356eb" (UID: "a8073975-c499-4cb6-ba86-9db9de1356eb"). InnerVolumeSpecName "kube-api-access-kx7cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:39.618644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.618613    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"]
Oct 31 00:15:39.619199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.619186    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a8073975-c499-4cb6-ba86-9db9de1356eb" containerName="revoke-keys"
Oct 31 00:15:39.619249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.619200    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8073975-c499-4cb6-ba86-9db9de1356eb" containerName="revoke-keys"
Oct 31 00:15:39.619309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.619300    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8073975-c499-4cb6-ba86-9db9de1356eb" containerName="revoke-keys"
Oct 31 00:15:39.623006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.622991    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.631957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.631930    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"]
Oct 31 00:15:39.643557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.636395    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8073975-c499-4cb6-ba86-9db9de1356eb-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:39.643557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.636424    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7cm\" (UniqueName: \"kubernetes.io/projected/a8073975-c499-4cb6-ba86-9db9de1356eb-kube-api-access-kx7cm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:39.738082 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.738048    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.738264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.738233    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tn6\" (UniqueName: \"kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.738307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.738283    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.839418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.839381    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.839844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.839813    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tn6\" (UniqueName: \"kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.839993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.839863    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.840776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.840755    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.843412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.843391    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.855707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.855676    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tn6\" (UniqueName: \"kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6\") pod \"maas-api-revoke-sub-cf64215d-rfcdv\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:39.952737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:39.952627    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:40.223712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.223672    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v" event={"ID":"a8073975-c499-4cb6-ba86-9db9de1356eb","Type":"ContainerDied","Data":"7425c675c6f0cbe83b010acbc49aa287599743b1ffb675ab6adceec72a9aebe8"}
Oct 31 00:15:40.223712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.223705    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"
Oct 31 00:15:40.223932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.223739    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7425c675c6f0cbe83b010acbc49aa287599743b1ffb675ab6adceec72a9aebe8"
Oct 31 00:15:40.225986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.225960    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq" event={"ID":"a168e155-83cf-4683-9e4e-a56681e0a24b","Type":"ContainerStarted","Data":"97f1f07c9013a57c52cf16322b0e3d3e8aae9ed7679a99ab01006c20d5902efb"}
Oct 31 00:15:40.226113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.226009    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq" event={"ID":"a168e155-83cf-4683-9e4e-a56681e0a24b","Type":"ContainerStarted","Data":"03933133da1b701aae96a819ab5734aa68c744a1e2694e08dde9c0d012919e3c"}
Oct 31 00:15:40.259815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.244048    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"]
Oct 31 00:15:40.259815 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:40.245029    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7bdebb_1b97_4515_8ac9_724bbae4f6c0.slice/crio-b94a5e8176dba44461ab6accd6bf6efc0ece8b3b71560a3a4272061f8d8ab00a WatchSource:0}: Error finding container b94a5e8176dba44461ab6accd6bf6efc0ece8b3b71560a3a4272061f8d8ab00a: Status 404 returned error can't find the container with id b94a5e8176dba44461ab6accd6bf6efc0ece8b3b71560a3a4272061f8d8ab00a
Oct 31 00:15:40.259815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.251250    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq" podStartSLOduration=2.251232354 podStartE2EDuration="2.251232354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:40.248320178 +0000 UTC m=+1308.194690896" watchObservedRunningTime="2026-10-31 00:15:40.251232354 +0000 UTC m=+1308.197603072"
Oct 31 00:15:40.522156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.522125    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"]
Oct 31 00:15:40.536581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.536554    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"]
Oct 31 00:15:40.536727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.536647    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.648903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.648860    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthwl\" (UniqueName: \"kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.649082 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.648912    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.649171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.649148    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.750412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.750308    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.750412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.750366    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthwl\" (UniqueName: \"kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.750412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.750413    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.751081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.751054    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.753080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.753058    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.769343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.769319    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthwl\" (UniqueName: \"kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl\") pod \"maas-api-revoke-sub-74dd79e9-tph8k\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:40.850957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:40.850923    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:41.169376 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:41.169329    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf1b1fa_8e6a_42f1_bfb4_4a75bbaa0b4a.slice/crio-831473f036f7ebddd5bd9fd580e72d667433a15bb7e9bb958020159c7881ff22 WatchSource:0}: Error finding container 831473f036f7ebddd5bd9fd580e72d667433a15bb7e9bb958020159c7881ff22: Status 404 returned error can't find the container with id 831473f036f7ebddd5bd9fd580e72d667433a15bb7e9bb958020159c7881ff22
Oct 31 00:15:41.170059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.170028    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"]
Oct 31 00:15:41.235037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.235002    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k" event={"ID":"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a","Type":"ContainerStarted","Data":"831473f036f7ebddd5bd9fd580e72d667433a15bb7e9bb958020159c7881ff22"}
Oct 31 00:15:41.237087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.237058    5078 generic.go:356] "Generic (PLEG): container finished" podID="a168e155-83cf-4683-9e4e-a56681e0a24b" containerID="97f1f07c9013a57c52cf16322b0e3d3e8aae9ed7679a99ab01006c20d5902efb" exitCode=0
Oct 31 00:15:41.237216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.237142    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq" event={"ID":"a168e155-83cf-4683-9e4e-a56681e0a24b","Type":"ContainerDied","Data":"97f1f07c9013a57c52cf16322b0e3d3e8aae9ed7679a99ab01006c20d5902efb"}
Oct 31 00:15:41.239209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.239178    5078 generic.go:356] "Generic (PLEG): container finished" podID="2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" containerID="6d9984a7acf26001621ec0d44b039c375cb9e3f4c941b8ec3becba2da21f6deb" exitCode=0
Oct 31 00:15:41.239411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.239392    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv" event={"ID":"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0","Type":"ContainerDied","Data":"6d9984a7acf26001621ec0d44b039c375cb9e3f4c941b8ec3becba2da21f6deb"}
Oct 31 00:15:41.239545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:41.239439    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv" event={"ID":"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0","Type":"ContainerStarted","Data":"b94a5e8176dba44461ab6accd6bf6efc0ece8b3b71560a3a4272061f8d8ab00a"}
Oct 31 00:15:42.248060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.248029    5078 generic.go:356] "Generic (PLEG): container finished" podID="ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" containerID="c290c619a5f1910efdaa33c233bb3c046a6b7d1da8ae60aaf6a8eb1664d11149" exitCode=0
Oct 31 00:15:42.248519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.248111    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k" event={"ID":"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a","Type":"ContainerDied","Data":"c290c619a5f1910efdaa33c233bb3c046a6b7d1da8ae60aaf6a8eb1664d11149"}
Oct 31 00:15:42.450702 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.450675    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:42.495296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.495259    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:42.581235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.581204    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6\" (UniqueName: \"kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6\") pod \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") "
Oct 31 00:15:42.581426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.581296    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca\") pod \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") "
Oct 31 00:15:42.581426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.581367    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca\") pod \"a168e155-83cf-4683-9e4e-a56681e0a24b\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") "
Oct 31 00:15:42.581802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.581778    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x\" (UniqueName: \"kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x\") pod \"a168e155-83cf-4683-9e4e-a56681e0a24b\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") "
Oct 31 00:15:42.581901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.581863    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca\") pod \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\" (UID: \"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0\") "
Oct 31 00:15:42.581960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.581921    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca\") pod \"a168e155-83cf-4683-9e4e-a56681e0a24b\" (UID: \"a168e155-83cf-4683-9e4e-a56681e0a24b\") "
Oct 31 00:15:42.583085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.583043    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca" pod "2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" (UID: "2e7bdebb-1b97-4515-8ac9-724bbae4f6c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:42.583172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.583085    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca" pod "a168e155-83cf-4683-9e4e-a56681e0a24b" (UID: "a168e155-83cf-4683-9e4e-a56681e0a24b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:42.585256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.585222    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x" pod "a168e155-83cf-4683-9e4e-a56681e0a24b" (UID: "a168e155-83cf-4683-9e4e-a56681e0a24b"). InnerVolumeSpecName "kube-api-access-9q28x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:42.586085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.586054    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca" pod "2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" (UID: "2e7bdebb-1b97-4515-8ac9-724bbae4f6c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:42.586565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.586540    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6" pod "2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" (UID: "2e7bdebb-1b97-4515-8ac9-724bbae4f6c0"). InnerVolumeSpecName "kube-api-access-z5tn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:42.587863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.587834    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca" pod "a168e155-83cf-4683-9e4e-a56681e0a24b" (UID: "a168e155-83cf-4683-9e4e-a56681e0a24b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:42.683829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.683794    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:42.683829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.683830    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q28x\" (UniqueName: \"kubernetes.io/projected/a168e155-83cf-4683-9e4e-a56681e0a24b-kube-api-access-9q28x\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:42.684108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.683844    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:42.684108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.683861    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e155-83cf-4683-9e4e-a56681e0a24b-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:42.684108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.683874    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tn6\" (UniqueName: \"kubernetes.io/projected/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-kube-api-access-z5tn6\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:42.684108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:42.683889    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:43.256105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.256065    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"
Oct 31 00:15:43.256565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.256064    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq" event={"ID":"a168e155-83cf-4683-9e4e-a56681e0a24b","Type":"ContainerDied","Data":"03933133da1b701aae96a819ab5734aa68c744a1e2694e08dde9c0d012919e3c"}
Oct 31 00:15:43.256565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.256199    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03933133da1b701aae96a819ab5734aa68c744a1e2694e08dde9c0d012919e3c"
Oct 31 00:15:43.257641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.257610    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv" event={"ID":"2e7bdebb-1b97-4515-8ac9-724bbae4f6c0","Type":"ContainerDied","Data":"b94a5e8176dba44461ab6accd6bf6efc0ece8b3b71560a3a4272061f8d8ab00a"}
Oct 31 00:15:43.257736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.257680    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b94a5e8176dba44461ab6accd6bf6efc0ece8b3b71560a3a4272061f8d8ab00a"
Oct 31 00:15:43.257736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.257625    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"
Oct 31 00:15:43.447255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.447198    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:43.497238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.497200    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca\") pod \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") "
Oct 31 00:15:43.497640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.497612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca" pod "ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" (UID: "ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:43.497759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.497722    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl\" (UniqueName: \"kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl\") pod \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") "
Oct 31 00:15:43.497918 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.497901    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca\") pod \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\" (UID: \"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a\") "
Oct 31 00:15:43.498379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.498357    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:43.500966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.500935    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca" pod "ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" (UID: "ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:43.501265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.501240    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl" pod "ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" (UID: "ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a"). InnerVolumeSpecName "kube-api-access-sthwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:43.599982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.599953    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:43.599982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:43.599981    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthwl\" (UniqueName: \"kubernetes.io/projected/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a-kube-api-access-sthwl\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:44.267887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:44.267848    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k" event={"ID":"ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a","Type":"ContainerDied","Data":"831473f036f7ebddd5bd9fd580e72d667433a15bb7e9bb958020159c7881ff22"}
Oct 31 00:15:44.268389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:44.267915    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831473f036f7ebddd5bd9fd580e72d667433a15bb7e9bb958020159c7881ff22"
Oct 31 00:15:44.268389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:44.267861    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"
Oct 31 00:15:47.811773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.811737    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"]
Oct 31 00:15:47.812710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812681    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a168e155-83cf-4683-9e4e-a56681e0a24b" containerName="revoke-keys"
Oct 31 00:15:47.812710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812706    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a168e155-83cf-4683-9e4e-a56681e0a24b" containerName="revoke-keys"
Oct 31 00:15:47.812890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812744    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" containerName="revoke-keys"
Oct 31 00:15:47.812890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812752    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" containerName="revoke-keys"
Oct 31 00:15:47.812890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812777    5078 cpu_manager.go:414] "containerMap: removing container" podUID="2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" containerName="revoke-keys"
Oct 31 00:15:47.812890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812785    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" containerName="revoke-keys"
Oct 31 00:15:47.813095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812917    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a168e155-83cf-4683-9e4e-a56681e0a24b" containerName="revoke-keys"
Oct 31 00:15:47.813095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812933    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" containerName="revoke-keys"
Oct 31 00:15:47.813095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.812945    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" containerName="revoke-keys"
Oct 31 00:15:47.818366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.818341    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.830533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.830506    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"]
Oct 31 00:15:47.880665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.880626    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.880665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.880668    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.880900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.880687    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.880968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.880939    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.881032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.881017    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.881086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.881059    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.881138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.881111    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2l48\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.881208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.881187    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.881263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.881220    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982330    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982382    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982416    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982433    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l48\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982503    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982541    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982613    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982644    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982663    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.982931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.982832    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.983427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.983396    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.983574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.983422    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.983574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.983457    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.984153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.984125    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.985807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.985785    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.986048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.986026    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.995961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.995940    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:47.996149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:47.996130    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l48\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:48.141000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.140967    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:48.187878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.186058    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"]
Oct 31 00:15:48.197961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.194648    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.246497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.236140    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"]
Oct 31 00:15:48.287257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287121    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287164    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287280    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2vn\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287315    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287347    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287371    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287515    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287560    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.287766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.287592    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.388734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.388701    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.388959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.388761    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.388959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.388810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2vn\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.388959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.388843    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.388959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.388876    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.388959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.388898    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389012    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389059    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389090    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389284    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389312    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389523    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389575    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.389941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.389876    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.391604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.391530    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.392215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.392190    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.407813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.407793    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2vn\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.409788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.409762    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.455117 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:48.455088    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1c1cec_fbce_4694_bb48_b6c32800f1e5.slice/crio-cc1114b1032cc9cd7b050c561cd8fd1d492f907277af9348467fc0a696438105 WatchSource:0}: Error finding container cc1114b1032cc9cd7b050c561cd8fd1d492f907277af9348467fc0a696438105: Status 404 returned error can't find the container with id cc1114b1032cc9cd7b050c561cd8fd1d492f907277af9348467fc0a696438105
Oct 31 00:15:48.456622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.456593    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"]
Oct 31 00:15:48.457903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.457868    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:48.457980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.457952    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:48.458018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.457995    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:48.546610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.546588    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:48.845797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.845768    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"]
Oct 31 00:15:48.845797 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:48.845778    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a113ee_02a0_456b_986c_1ad881dc630d.slice/crio-d1bb0848c206e7094b7c8738b4728a7975cf2e260ef36b967f4e1e9e210da8c0 WatchSource:0}: Error finding container d1bb0848c206e7094b7c8738b4728a7975cf2e260ef36b967f4e1e9e210da8c0: Status 404 returned error can't find the container with id d1bb0848c206e7094b7c8738b4728a7975cf2e260ef36b967f4e1e9e210da8c0
Oct 31 00:15:48.848262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.848231    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:48.848368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.848306    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:48.848368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:48.848349    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:49.342884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.342844    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" event={"ID":"c7a113ee-02a0-456b-986c-1ad881dc630d","Type":"ContainerStarted","Data":"ef354a33f057b7b351508442e65d49eda49b50627ba9efb50cb1453ac27b6554"}
Oct 31 00:15:49.343046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.342909    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" event={"ID":"c7a113ee-02a0-456b-986c-1ad881dc630d","Type":"ContainerStarted","Data":"d1bb0848c206e7094b7c8738b4728a7975cf2e260ef36b967f4e1e9e210da8c0"}
Oct 31 00:15:49.344756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.344727    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" event={"ID":"1f1c1cec-fbce-4694-bb48-b6c32800f1e5","Type":"ContainerStarted","Data":"00a6f8ab94bfb3611cdf3d8da9eb7e571e9a8d74b02c7f8f94aa31d61012cdb3"}
Oct 31 00:15:49.344919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.344774    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" event={"ID":"1f1c1cec-fbce-4694-bb48-b6c32800f1e5","Type":"ContainerStarted","Data":"cc1114b1032cc9cd7b050c561cd8fd1d492f907277af9348467fc0a696438105"}
Oct 31 00:15:49.406499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.398477    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podStartSLOduration=1.398431132 podStartE2EDuration="1.398431132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:49.374788506 +0000 UTC m=+1317.321159225" watchObservedRunningTime="2026-10-31 00:15:49.398431132 +0000 UTC m=+1317.344801851"
Oct 31 00:15:49.406499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.403865    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podStartSLOduration=2.403845275 podStartE2EDuration="2.403845275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:49.401081799 +0000 UTC m=+1317.347452531" watchObservedRunningTime="2026-10-31 00:15:49.403845275 +0000 UTC m=+1317.350215993"
Oct 31 00:15:49.547125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.547093    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:49.548982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.548956    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:49.549121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.549007    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:49.898497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.897550    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"]
Oct 31 00:15:49.919177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.902932    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:49.944388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.943151    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:49.944388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.943924    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:15:49.946366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:49.946269    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"]
Oct 31 00:15:50.042491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.021900    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.042897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.042863    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.043072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.043057    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdrf\" (UniqueName: \"kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.141594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.141443    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:50.143034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.143012    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:15:50.143102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.143054    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:15:50.144059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.143808    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.144059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.143852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.144059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.143983    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdrf\" (UniqueName: \"kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.144819 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.144802    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.146681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.146647    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.159517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.159438    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdrf\" (UniqueName: \"kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf\") pod \"maas-api-revoke-sub-72b901e3-7ndxj\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.266497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.266444    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:50.561226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.560382    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:50.563281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.563245    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:50.864566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.864487    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"]
Oct 31 00:15:50.886496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.875717    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:50.896709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.896354    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"]
Oct 31 00:15:50.954658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:50.954519    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"]
Oct 31 00:15:51.064056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.064031    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpr9w\" (UniqueName: \"kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.064152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.064072    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.064285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.064273    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.142387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.142346    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:15:51.142603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.142407    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:15:51.165514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.165449    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpr9w\" (UniqueName: \"kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.165636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.165545    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.165737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.165720    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.166369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.166346    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.234343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.234308    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.235394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.235361    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpr9w\" (UniqueName: \"kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w\") pod \"maas-api-revoke-sub-40a05f44-4pbv5\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.257556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.257527    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:51.370273 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.370239    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj" event={"ID":"0e3f2de8-172d-4c9d-9e29-4ca873325531","Type":"ContainerStarted","Data":"adfc4629370d6f367daf10c77f2dc42d1bbc0b4a60c84b29b9ea82c5260e37aa"}
Oct 31 00:15:51.550071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.550035    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:51.550302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.550100    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:51.819891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:51.819850    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"]
Oct 31 00:15:52.141687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.141648    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:15:52.142139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.141700    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:15:52.444916 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.444187    5078 generic.go:356] "Generic (PLEG): container finished" podID="0e3f2de8-172d-4c9d-9e29-4ca873325531" containerID="d9937420a2d46c2985a562ad0eb0d974743b754314359e0e8a5943220d14ac3d" exitCode=0
Oct 31 00:15:52.448526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.444938    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj" event={"ID":"0e3f2de8-172d-4c9d-9e29-4ca873325531","Type":"ContainerDied","Data":"d9937420a2d46c2985a562ad0eb0d974743b754314359e0e8a5943220d14ac3d"}
Oct 31 00:15:52.450731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.450707    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5" event={"ID":"45db12f8-975f-45c3-a7e1-064f7135d6e1","Type":"ContainerStarted","Data":"98a8f7ca06567f68dba1549be63fa2ca07691c3bdfca673d0c33e1d8d946ee07"}
Oct 31 00:15:52.547397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.547362    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:52.547601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:52.547413    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:53.144039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.143517    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:15:53.144039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.143589    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:15:53.460222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.460186    5078 generic.go:356] "Generic (PLEG): container finished" podID="45db12f8-975f-45c3-a7e1-064f7135d6e1" containerID="4ccb4a61c84ddad894325c1094774773b05700bd68c70bf4618aa4c2ebafe4c2" exitCode=0
Oct 31 00:15:53.460728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.460703    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5" event={"ID":"45db12f8-975f-45c3-a7e1-064f7135d6e1","Type":"ContainerDied","Data":"4ccb4a61c84ddad894325c1094774773b05700bd68c70bf4618aa4c2ebafe4c2"}
Oct 31 00:15:53.547222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.547179    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:53.547394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.547241    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:53.934775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:53.934738    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:54.038946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.038545    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf\" (UniqueName: \"kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf\") pod \"0e3f2de8-172d-4c9d-9e29-4ca873325531\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") "
Oct 31 00:15:54.038946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.038702    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca\") pod \"0e3f2de8-172d-4c9d-9e29-4ca873325531\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") "
Oct 31 00:15:54.039209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.039023    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca\") pod \"0e3f2de8-172d-4c9d-9e29-4ca873325531\" (UID: \"0e3f2de8-172d-4c9d-9e29-4ca873325531\") "
Oct 31 00:15:54.040293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.040237    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca" pod "0e3f2de8-172d-4c9d-9e29-4ca873325531" (UID: "0e3f2de8-172d-4c9d-9e29-4ca873325531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:54.045733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.045647    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca" pod "0e3f2de8-172d-4c9d-9e29-4ca873325531" (UID: "0e3f2de8-172d-4c9d-9e29-4ca873325531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:54.063989 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.063951    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf" pod "0e3f2de8-172d-4c9d-9e29-4ca873325531" (UID: "0e3f2de8-172d-4c9d-9e29-4ca873325531"). InnerVolumeSpecName "kube-api-access-jrdrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:54.143503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.140970    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:54.143503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.141005    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrdrf\" (UniqueName: \"kubernetes.io/projected/0e3f2de8-172d-4c9d-9e29-4ca873325531-kube-api-access-jrdrf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:54.143503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.141022    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3f2de8-172d-4c9d-9e29-4ca873325531-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:54.143503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.141319    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:15:54.143503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.141367    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:15:54.468112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.468033    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"
Oct 31 00:15:54.468582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.468553    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj" event={"ID":"0e3f2de8-172d-4c9d-9e29-4ca873325531","Type":"ContainerDied","Data":"adfc4629370d6f367daf10c77f2dc42d1bbc0b4a60c84b29b9ea82c5260e37aa"}
Oct 31 00:15:54.468743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.468729    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adfc4629370d6f367daf10c77f2dc42d1bbc0b4a60c84b29b9ea82c5260e37aa"
Oct 31 00:15:54.547074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.547046    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:54.547364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.547335    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:54.755810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.755788    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:54.757815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.757794    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca\") pod \"45db12f8-975f-45c3-a7e1-064f7135d6e1\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") "
Oct 31 00:15:54.757999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.757984    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w\" (UniqueName: \"kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w\") pod \"45db12f8-975f-45c3-a7e1-064f7135d6e1\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") "
Oct 31 00:15:54.758073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.758058    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca\") pod \"45db12f8-975f-45c3-a7e1-064f7135d6e1\" (UID: \"45db12f8-975f-45c3-a7e1-064f7135d6e1\") "
Oct 31 00:15:54.758134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.758110    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca" pod "45db12f8-975f-45c3-a7e1-064f7135d6e1" (UID: "45db12f8-975f-45c3-a7e1-064f7135d6e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:54.758484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.758449    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:54.760388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.760369    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w" pod "45db12f8-975f-45c3-a7e1-064f7135d6e1" (UID: "45db12f8-975f-45c3-a7e1-064f7135d6e1"). InnerVolumeSpecName "kube-api-access-qpr9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:54.760669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.760654    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca" pod "45db12f8-975f-45c3-a7e1-064f7135d6e1" (UID: "45db12f8-975f-45c3-a7e1-064f7135d6e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:54.860170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.860145    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpr9w\" (UniqueName: \"kubernetes.io/projected/45db12f8-975f-45c3-a7e1-064f7135d6e1-kube-api-access-qpr9w\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:54.860170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:54.860173    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db12f8-975f-45c3-a7e1-064f7135d6e1-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:55.141662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.141632    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:15:55.141864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.141688    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:15:55.488542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.483712    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5" event={"ID":"45db12f8-975f-45c3-a7e1-064f7135d6e1","Type":"ContainerDied","Data":"98a8f7ca06567f68dba1549be63fa2ca07691c3bdfca673d0c33e1d8d946ee07"}
Oct 31 00:15:55.488542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.483778    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98a8f7ca06567f68dba1549be63fa2ca07691c3bdfca673d0c33e1d8d946ee07"
Oct 31 00:15:55.488542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.483853    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"
Oct 31 00:15:55.547240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.547193    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:15:55.547394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:55.547271    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:15:56.147014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.146981    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:56.147431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.147411    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:56.148176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.148160    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:15:56.323100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.323063    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"]
Oct 31 00:15:56.324579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.324175    5078 cpu_manager.go:414] "containerMap: removing container" podUID="45db12f8-975f-45c3-a7e1-064f7135d6e1" containerName="revoke-keys"
Oct 31 00:15:56.324579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.324200    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45db12f8-975f-45c3-a7e1-064f7135d6e1" containerName="revoke-keys"
Oct 31 00:15:56.324579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.324223    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0e3f2de8-172d-4c9d-9e29-4ca873325531" containerName="revoke-keys"
Oct 31 00:15:56.324579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.324230    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3f2de8-172d-4c9d-9e29-4ca873325531" containerName="revoke-keys"
Oct 31 00:15:56.324579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.324399    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="45db12f8-975f-45c3-a7e1-064f7135d6e1" containerName="revoke-keys"
Oct 31 00:15:56.324579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.324414    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3f2de8-172d-4c9d-9e29-4ca873325531" containerName="revoke-keys"
Oct 31 00:15:56.330490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.330440    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.346808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.346775    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"distinct-w2-7c5da3-kserve-self-signed-certs\""
Oct 31 00:15:56.366561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.366531    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"]
Oct 31 00:15:56.395256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.395099    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.395256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.395181    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.395256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.395234    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnsn\" (UniqueName: \"kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.395566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.395288    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.395566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.395349    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.395566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.395410    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.496219    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.496310    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.496456    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.496572    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.496622    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnsn\" (UniqueName: \"kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.496668    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.497542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.497193    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.498314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.497550    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.498314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.497767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.508522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.501159    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.514017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.511718    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.519256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.519201    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnsn\" (UniqueName: \"kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn\") pod \"distinct-w2-7c5da3-kserve-6c4b446d88-pzqww\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:56.552650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.552617    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:56.553028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.553006    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:56.553876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.553856    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:15:56.705920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:56.705878    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:15:57.020651 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:15:57.020582    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f13311b_d0f7_4d27_96ee_7e6eca986f43.slice/crio-f68f264c4304063c856cef065f35200784e29c41e74e94a1e18b29a714098bfb WatchSource:0}: Error finding container f68f264c4304063c856cef065f35200784e29c41e74e94a1e18b29a714098bfb: Status 404 returned error can't find the container with id f68f264c4304063c856cef065f35200784e29c41e74e94a1e18b29a714098bfb
Oct 31 00:15:57.021019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:57.020987    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"]
Oct 31 00:15:57.502631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:57.502591    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" event={"ID":"6f13311b-d0f7-4d27-96ee-7e6eca986f43","Type":"ContainerStarted","Data":"56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406"}
Oct 31 00:15:57.503068 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:57.502641    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" event={"ID":"6f13311b-d0f7-4d27-96ee-7e6eca986f43","Type":"ContainerStarted","Data":"f68f264c4304063c856cef065f35200784e29c41e74e94a1e18b29a714098bfb"}
Oct 31 00:15:57.531306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:57.531246    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podStartSLOduration=1.5312310519999999 podStartE2EDuration="1.531231052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:57.529440269 +0000 UTC m=+1325.475810984" watchObservedRunningTime="2026-10-31 00:15:57.531231052 +0000 UTC m=+1325.477601768"
Oct 31 00:15:57.706895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:15:57.706860    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:04.786622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:04.786576    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:04.977162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:04.977104    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:05.175595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:05.174571    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:05.175894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:05.175867    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:06.390733 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:16:06.390690    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:16:06.707011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:06.706920    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:06.723759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:06.723731    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:06.797985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:06.797952    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:12.504858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.504817    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"]
Oct 31 00:16:12.510656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.510632    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.519970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.519936    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"default-dockercfg-r8fdg\""
Oct 31 00:16:12.520145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.519969    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"kube-root-ca.crt\""
Oct 31 00:16:12.520145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.520025    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"openshift-service-ca.crt\""
Oct 31 00:16:12.521167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.521095    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"]
Oct 31 00:16:12.616250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.616222    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"distinct-w0-c60826-kserve-self-signed-certs\""
Oct 31 00:16:12.680353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.680311    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.680581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.680365    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.680581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.680437    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.680697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.680599    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhjd\" (UniqueName: \"kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.680697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.680686    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.680810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.680772    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.782386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.782283    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.782665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.782417    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.782665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.782452    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.782665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.782539    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.782665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.782593    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhjd\" (UniqueName: \"kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.782884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.782681    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.783108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.783051    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.783178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.783121    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.792571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.783611    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.792571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.785446    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.792571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.791984    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.803962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.803933    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhjd\" (UniqueName: \"kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd\") pod \"distinct-w0-c60826-kserve-8497468c4d-s54tw\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:12.833267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:12.833234    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:13.151048 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:13.151004    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b68920b_68ef_4980_a743_f46eddf82cf5.slice/crio-aed8aa631bce388726da4f6fdbd5c770a8775b6778bfce806f7cf35931e36782 WatchSource:0}: Error finding container aed8aa631bce388726da4f6fdbd5c770a8775b6778bfce806f7cf35931e36782: Status 404 returned error can't find the container with id aed8aa631bce388726da4f6fdbd5c770a8775b6778bfce806f7cf35931e36782
Oct 31 00:16:13.152434 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:13.152406    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"]
Oct 31 00:16:13.837017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:13.836974    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" event={"ID":"9b68920b-68ef-4980-a743-f46eddf82cf5","Type":"ContainerStarted","Data":"ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b"}
Oct 31 00:16:13.837017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:13.837023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" event={"ID":"9b68920b-68ef-4980-a743-f46eddf82cf5","Type":"ContainerStarted","Data":"aed8aa631bce388726da4f6fdbd5c770a8775b6778bfce806f7cf35931e36782"}
Oct 31 00:16:13.865682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:13.865624    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podStartSLOduration=1.865610124 podStartE2EDuration="1.865610124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:13.86155065 +0000 UTC m=+1341.807921367" watchObservedRunningTime="2026-10-31 00:16:13.865610124 +0000 UTC m=+1341.811980874"
Oct 31 00:16:14.834015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:14.833965    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:17.550008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.549974    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"]
Oct 31 00:16:17.554650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.554629    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.564248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.563911    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7b2341-openshift-default-dockercfg-7tk5j\""
Oct 31 00:16:17.566902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.566878    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"]
Oct 31 00:16:17.645325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645292    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmd8\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645338    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645439    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645613    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645658    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645728    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645766    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.645901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.645856    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.647184 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.647156    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"]
Oct 31 00:16:17.652052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.652037    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.663698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.663675    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"]
Oct 31 00:16:17.746737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.746700    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.746920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.746761    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.746920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.746815    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747013    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747061    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmd8\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747134    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747219    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747261    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747303    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747331    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747359    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgn9\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747413    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747439    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747604    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747653    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747609    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747692    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.747836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747807    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.747836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747828    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.748002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747849    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.748002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747905    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.748002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.747926    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.748167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.748151    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.749789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.749763    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.750014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.749996    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.760270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.760250    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.760384 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.760338    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmd8\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.848955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.848767    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.848955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.848813    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.848955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.848842    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgn9\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.848955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.848898    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.848955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.848924    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.849337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.848976    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.849337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.849015    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.849337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.849135    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.849337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.849235    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.851290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.851262    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"]
Oct 31 00:16:17.852394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.852364    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.852712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.852684    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.852815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.852797    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.852964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.852945    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.853040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.853018    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.860632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.858166    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.860632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.859074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.860632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.860354    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.882883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.877704    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"]
Oct 31 00:16:17.886060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.885616    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.888547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.888501    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgn9\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:17.896035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.895995    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:17.950483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950432    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950524    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwd2\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950569    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950617    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950648    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950698    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950749    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950764    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.950870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.950819    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:17.981792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:17.981536    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:18.051938 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.051889    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.051944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwd2\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.051979    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.052034    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.052073    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.052137    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.052198    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.052224    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.052581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.052292    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.054111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.053318    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.054111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.053378    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.054111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.053717    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.054111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.053980    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.054631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.054270    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.059071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.059005    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.059200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.059059    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.066534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.066286    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwd2\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.066997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.066975    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.204212 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:18.204170    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8de2cb_4f63_4998_93a9_aff00d53183e.slice/crio-d382f98c0b597c4281d978892d1a874df9c169480b1f03be0833c29c6383e38e WatchSource:0}: Error finding container d382f98c0b597c4281d978892d1a874df9c169480b1f03be0833c29c6383e38e: Status 404 returned error can't find the container with id d382f98c0b597c4281d978892d1a874df9c169480b1f03be0833c29c6383e38e
Oct 31 00:16:18.205929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.205894    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"]
Oct 31 00:16:18.207243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.207204    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.207316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.207290    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.207356 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.207332    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.228753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.228735    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:18.289156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.288952    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"]
Oct 31 00:16:18.300948 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:18.299980    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec42ad3_37b6_41b2_930b_5756d51a23b3.slice/crio-01f410f494180c9195d240e3d4da65d932515e7d553117ebff1b455bb0fcaefb WatchSource:0}: Error finding container 01f410f494180c9195d240e3d4da65d932515e7d553117ebff1b455bb0fcaefb: Status 404 returned error can't find the container with id 01f410f494180c9195d240e3d4da65d932515e7d553117ebff1b455bb0fcaefb
Oct 31 00:16:18.302990 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.302957    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.303107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.303028    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.303107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.303062    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.539277 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:18.539238    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad9fb99_b725_4f80_b139_ceaba025da2c.slice/crio-5b721aefa8592dd83d269687c527113174a534c3141e53dd37279666eb0f22b2 WatchSource:0}: Error finding container 5b721aefa8592dd83d269687c527113174a534c3141e53dd37279666eb0f22b2: Status 404 returned error can't find the container with id 5b721aefa8592dd83d269687c527113174a534c3141e53dd37279666eb0f22b2
Oct 31 00:16:18.540072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.540048    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"]
Oct 31 00:16:18.542788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.542754    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.542872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.542826    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.542912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.542869    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:18.884177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.884080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" event={"ID":"3ad9fb99-b725-4f80-b139-ceaba025da2c","Type":"ContainerStarted","Data":"71d1bd195037b7c88fa1334098ca04cdd7de630643d9b3101d54cc75ab1f7338"}
Oct 31 00:16:18.884811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.884783    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" event={"ID":"3ad9fb99-b725-4f80-b139-ceaba025da2c","Type":"ContainerStarted","Data":"5b721aefa8592dd83d269687c527113174a534c3141e53dd37279666eb0f22b2"}
Oct 31 00:16:18.888679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.888652    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" event={"ID":"0ec42ad3-37b6-41b2-930b-5756d51a23b3","Type":"ContainerStarted","Data":"715907364b15dc74f4b24741a16ee080ca2da1326f021eccf9dee409973fd1e0"}
Oct 31 00:16:18.888862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.888844    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" event={"ID":"0ec42ad3-37b6-41b2-930b-5756d51a23b3","Type":"ContainerStarted","Data":"01f410f494180c9195d240e3d4da65d932515e7d553117ebff1b455bb0fcaefb"}
Oct 31 00:16:18.892286 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.892252    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" event={"ID":"aa8de2cb-4f63-4998-93a9-aff00d53183e","Type":"ContainerStarted","Data":"253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413"}
Oct 31 00:16:18.892402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.892307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" event={"ID":"aa8de2cb-4f63-4998-93a9-aff00d53183e","Type":"ContainerStarted","Data":"d382f98c0b597c4281d978892d1a874df9c169480b1f03be0833c29c6383e38e"}
Oct 31 00:16:18.899093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.899068    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:18.917000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.916932    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podStartSLOduration=1.916914662 podStartE2EDuration="1.916914662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:18.913713049 +0000 UTC m=+1346.860083793" watchObservedRunningTime="2026-10-31 00:16:18.916914662 +0000 UTC m=+1346.863285380"
Oct 31 00:16:18.945941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.945885    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podStartSLOduration=1.945865846 podStartE2EDuration="1.945865846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:18.940054521 +0000 UTC m=+1346.886425257" watchObservedRunningTime="2026-10-31 00:16:18.945865846 +0000 UTC m=+1346.892236565"
Oct 31 00:16:18.970426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.970370    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podStartSLOduration=1.9703529309999999 podStartE2EDuration="1.970352931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:18.966671425 +0000 UTC m=+1346.913042155" watchObservedRunningTime="2026-10-31 00:16:18.970352931 +0000 UTC m=+1346.916723648"
Oct 31 00:16:18.985725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:18.985694    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:19.236115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:19.236082    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:19.238389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:19.237893    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:19.238389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:19.237949    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:19.910417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:19.910383    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:19.910993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:19.910960    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:20.018936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:20.018897    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:20.019266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:20.019233    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:20.241199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:20.241104    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:20.241481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:20.241427    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:20.900063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:20.900026    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:20.900376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:20.900349    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:21.009100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.009066    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:21.009700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.009668    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:21.275552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.251231    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:21.275552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.251302    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:21.903594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.903557    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:21.926440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.926396    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:22.001526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.992108    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:22.001526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:21.992168    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:22.242546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.232290    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:22.242546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.232381    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:22.701618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.701563    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:16:22.781652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.781611    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:22.794204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.794163    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:22.835451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.834756    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:22.911919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.911882    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:22.912201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.912168    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:22.915781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.915698    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:22.983103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.983003    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:22.985215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.985184    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:22.985414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:22.985391    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:23.176673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.176611    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:23.229926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.229882    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:23.230112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.229950    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:23.568847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.568797    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:23.659615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.658961    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:16:23.702645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.702597    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:23.703399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.703063    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:16:23.890336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.890295    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:23.891228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.891203    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:23.908485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.903971    5078 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:23.908485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.907059    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:23.921862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.909595    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:23.921862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.909636    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:23.921862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.909710    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:16:23.921862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.919522    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:16:23.921862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:23.919863    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:16:24.086500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.059710    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:16:24.146205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.146120    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:16:24.181590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.181546    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:16:24.182264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.182240    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:16:24.234865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.234573    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:24.234865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.234627    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:24.246569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.240029    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:24.246569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.240076    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:24.446825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.446733    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:24.569321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.569279    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:24.592222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.592172    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:16:24.592896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.592875    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:16:24.643488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.640545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:24.669488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.659580    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:24.669488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.659994    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:16:24.767586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.767488    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 31 00:16:24.768074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.768039    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:24.907537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.907493    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:24.911305 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.910339    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:24.919882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.919856    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:24.920113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.919783    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:24.989786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.989751    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:24.990028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:24.989997    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:25.147855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.147806    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.148369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.148341    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.187822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.187776    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.240593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.240547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.304361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.302847    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:25.304361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.302902    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:25.597547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.555758    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:16:25.597547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.596549    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.664876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.664826    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.687484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.671584    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:16:25.687484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.665132    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.746489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.727211    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:16:25.938549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.912545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:16:25.942485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.938806    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:25.959719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.959644    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:25.960039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:25.959965    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:26.030498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.014588    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:26.030498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.014652    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:26.148752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.148712    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:26.236355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.236268    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:26.236616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.236590    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:26.269600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.269216    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:26.398525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.398496    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:26.400290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.399208    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:16:26.687674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.687637    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:16:26.707970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.707932    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:26.756507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.725801    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:26.806486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.805780    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:16:26.920872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.918654    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:16:26.924486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:26.921132    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 00:16:27.067143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.067053    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:16:27.067419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.067403    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:16:27.068824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.068729    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:16:27.174294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.174217    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:16:27.174947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.174908    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:16:27.229604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.229572    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:16:27.266770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.242684    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:16:27.267024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:27.266996    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:16:28.688412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.688370    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:28.690320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.690299    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:28.692095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.692070    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:16:28.784363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.784326    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:28.786658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.786637    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:28.798983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.798952    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:16:28.943771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.943686    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:16:28.944037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:28.943983    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:29.295791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:29.295701    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:29.296351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:29.296294    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:29.298322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:29.298304    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:16:31.439179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.439144    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"]
Oct 31 00:16:31.445634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.445613    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.458799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.458770    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"distinct-2-w6-0402df-kserve-self-signed-certs\""
Oct 31 00:16:31.464577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.464557    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"]
Oct 31 00:16:31.498220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.498188    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.498220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.498224    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.498426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.498254    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szx2g\" (UniqueName: \"kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.498426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.498379    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.498582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.498426    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.498582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.498541    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.625262    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.625349    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626134    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626166    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626213    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szx2g\" (UniqueName: \"kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626271    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626628    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.626932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.626816    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.652485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.650899    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szx2g\" (UniqueName: \"kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.663064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.663031    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.664594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.664568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs\") pod \"distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:31.763931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:31.763853    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:32.067074 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:32.067038    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ef0b0ba_12f4_4405_89ad_fcc33ffa23fd.slice/crio-1a026f967063b42c914cb7de419c673c15873021374222e2b22eb910a14554ce WatchSource:0}: Error finding container 1a026f967063b42c914cb7de419c673c15873021374222e2b22eb910a14554ce: Status 404 returned error can't find the container with id 1a026f967063b42c914cb7de419c673c15873021374222e2b22eb910a14554ce
Oct 31 00:16:32.067344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.067322    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"]
Oct 31 00:16:32.272187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.272151    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" event={"ID":"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd","Type":"ContainerStarted","Data":"80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6"}
Oct 31 00:16:32.272187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.272194    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" event={"ID":"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd","Type":"ContainerStarted","Data":"1a026f967063b42c914cb7de419c673c15873021374222e2b22eb910a14554ce"}
Oct 31 00:16:32.332121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.332028    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podStartSLOduration=1.332014389 podStartE2EDuration="1.332014389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:32.330127921 +0000 UTC m=+1360.276498621" watchObservedRunningTime="2026-10-31 00:16:32.332014389 +0000 UTC m=+1360.278385133"
Oct 31 00:16:32.763964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.763920    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:32.846140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.846110    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:32.858445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:32.858422    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:33.237507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.237478    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"]
Oct 31 00:16:33.241528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.241506    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.283194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.283168    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"]
Oct 31 00:16:33.361062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361027    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361097    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361129    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361277    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361329    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5nqj\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361432    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361508    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361555    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.361748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.361729    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463288    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5nqj\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463373    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463437    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463507    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463530    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463730    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463802    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.463947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.463896    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.464093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.464070    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.464335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.464191    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.464335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.464275    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.464519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.464500    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.464591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.464574    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.465992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.465968    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.466864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.466847    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.482635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.482610    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5nqj\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.484735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.484716    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.577193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.577098    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:33.876380 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:33.876348    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc916afdf_77b9_4ffc_8c9e_3b89f67269b8.slice/crio-062293a84fd2c7da09ccf41d3c4aa2173733be8c4b08de5046fb61c0a4eb9caa WatchSource:0}: Error finding container 062293a84fd2c7da09ccf41d3c4aa2173733be8c4b08de5046fb61c0a4eb9caa: Status 404 returned error can't find the container with id 062293a84fd2c7da09ccf41d3c4aa2173733be8c4b08de5046fb61c0a4eb9caa
Oct 31 00:16:33.878285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.878259    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"]
Oct 31 00:16:33.879780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.879740    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:33.879866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.879832    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:33.879917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:33.879876    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:34.296399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:34.296361    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" event={"ID":"c916afdf-77b9-4ffc-8c9e-3b89f67269b8","Type":"ContainerStarted","Data":"e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84"}
Oct 31 00:16:34.296597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:34.296422    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" event={"ID":"c916afdf-77b9-4ffc-8c9e-3b89f67269b8","Type":"ContainerStarted","Data":"062293a84fd2c7da09ccf41d3c4aa2173733be8c4b08de5046fb61c0a4eb9caa"}
Oct 31 00:16:34.328859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:34.328804    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podStartSLOduration=1.328789985 podStartE2EDuration="1.328789985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:34.324677521 +0000 UTC m=+1362.271048240" watchObservedRunningTime="2026-10-31 00:16:34.328789985 +0000 UTC m=+1362.275160702"
Oct 31 00:16:34.595550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:34.594824    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:34.602575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:34.601588    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:34.602575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:34.601657    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:35.589149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:35.589117    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:35.589778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:35.589746    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:36.431392 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:16:36.429871    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:16:36.581514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:36.579422    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:36.581514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:36.579503    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:37.583486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:37.578822    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:37.583486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:37.578903    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:38.061873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.061836    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"]
Oct 31 00:16:38.117484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.111659    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"]
Oct 31 00:16:38.117484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.111786    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.126515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.125699    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"distinct-2-w1-4c360e-kserve-self-signed-certs\""
Oct 31 00:16:38.267331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.267273    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.267574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.267401    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.267574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.267503    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.267574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.267538    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gkm\" (UniqueName: \"kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.267787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.267700    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.276508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.267883    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.369713    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.369846    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.369884    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.369974    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.370070    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.370113    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gkm\" (UniqueName: \"kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.370844    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.371069    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.387543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.371275    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.415425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.398163    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.415425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.415222    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gkm\" (UniqueName: \"kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.415947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.415921    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs\") pod \"distinct-2-w1-4c360e-kserve-759c984544-2stvb\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.434816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.434794    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:38.608257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.608147    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:38.608257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.608230    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:38.760966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:38.760938    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"]
Oct 31 00:16:38.761098 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:38.760962    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod869264e1_101e_4adb_b8f6_17bfa992b217.slice/crio-7820824ab8f94a2229586d57f96b261e3fb66f7e1f63a6509726f9d0293ed67d WatchSource:0}: Error finding container 7820824ab8f94a2229586d57f96b261e3fb66f7e1f63a6509726f9d0293ed67d: Status 404 returned error can't find the container with id 7820824ab8f94a2229586d57f96b261e3fb66f7e1f63a6509726f9d0293ed67d
Oct 31 00:16:39.348587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:39.348549    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" event={"ID":"869264e1-101e-4adb-b8f6-17bfa992b217","Type":"ContainerStarted","Data":"cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1"}
Oct 31 00:16:39.348795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:39.348598    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" event={"ID":"869264e1-101e-4adb-b8f6-17bfa992b217","Type":"ContainerStarted","Data":"7820824ab8f94a2229586d57f96b261e3fb66f7e1f63a6509726f9d0293ed67d"}
Oct 31 00:16:39.412300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:39.412015    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podStartSLOduration=2.411995815 podStartE2EDuration="2.411995815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:39.386561824 +0000 UTC m=+1367.332932542" watchObservedRunningTime="2026-10-31 00:16:39.411995815 +0000 UTC m=+1367.358366531"
Oct 31 00:16:39.436099 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:39.436067    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:39.584494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:39.581800    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:39.584494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:39.581857    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:40.577998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:40.577961    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 00:16:40.578548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:40.578021    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 00:16:41.584334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:41.584303    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:41.584877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:41.584596    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:41.598645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:41.598610    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:16:41.764284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:41.764255    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:41.777358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:41.777330    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:42.395060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:42.395027    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:43.304774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.304736    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"]
Oct 31 00:16:43.308970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.308945    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.317274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.317252    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:16:43.317531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.317514    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:16:43.321725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.321492    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"]
Oct 31 00:16:43.402185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.402152    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wg2c\" (UniqueName: \"kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.402438 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.402419    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.402592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.402570    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.504538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.504502    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.504785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.504613    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.504785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.504695    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wg2c\" (UniqueName: \"kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.505399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.505285    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.507628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.507603    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.518767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.518742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wg2c\" (UniqueName: \"kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c\") pod \"maas-api-revoke-sub-db9d5bbe-m2d7m\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.629236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.629140    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:43.933174 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:43.933142    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656b98cc_a8be_402e_ac4e_a90ec2f12c38.slice/crio-651bc02ec3f6774c6b1c31a73b8d981ee4b8edccbe30f00d2d3ef423b4d4a564 WatchSource:0}: Error finding container 651bc02ec3f6774c6b1c31a73b8d981ee4b8edccbe30f00d2d3ef423b4d4a564: Status 404 returned error can't find the container with id 651bc02ec3f6774c6b1c31a73b8d981ee4b8edccbe30f00d2d3ef423b4d4a564
Oct 31 00:16:43.933479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:43.933441    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"]
Oct 31 00:16:44.426947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:44.426892    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m" event={"ID":"656b98cc-a8be-402e-ac4e-a90ec2f12c38","Type":"ContainerStarted","Data":"e9348376250757d9b8aac40faa34b41ea15b689900b61bbe51954124e74596b0"}
Oct 31 00:16:44.427507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:44.426991    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m" event={"ID":"656b98cc-a8be-402e-ac4e-a90ec2f12c38","Type":"ContainerStarted","Data":"651bc02ec3f6774c6b1c31a73b8d981ee4b8edccbe30f00d2d3ef423b4d4a564"}
Oct 31 00:16:44.453443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:44.453348    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m" podStartSLOduration=1.453334688 podStartE2EDuration="1.453334688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:44.4474861 +0000 UTC m=+1372.393856817" watchObservedRunningTime="2026-10-31 00:16:44.453334688 +0000 UTC m=+1372.399705447"
Oct 31 00:16:45.484487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:45.483446    5078 generic.go:356] "Generic (PLEG): container finished" podID="656b98cc-a8be-402e-ac4e-a90ec2f12c38" containerID="e9348376250757d9b8aac40faa34b41ea15b689900b61bbe51954124e74596b0" exitCode=0
Oct 31 00:16:45.484487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:45.483521    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m" event={"ID":"656b98cc-a8be-402e-ac4e-a90ec2f12c38","Type":"ContainerDied","Data":"e9348376250757d9b8aac40faa34b41ea15b689900b61bbe51954124e74596b0"}
Oct 31 00:16:46.333487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.332551    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:46.572640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.566561    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:46.617546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.614512    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:16:46.730490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.728571    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:46.835484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.829549    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:46.867350 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.866684    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:16:46.924488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.921577    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:46.924488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:46.922866    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.287546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.285718    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:16:47.340498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.334995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.340498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.335669    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:16:47.378575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.378514    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.399503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.392502    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.644687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.578790    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.644687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.595641    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:16:47.644687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.608016    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:16:47.788868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.761669    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.788868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.762068    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:16:47.788868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.762355    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 31 00:16:47.788868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.762485    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:16:47.788868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.762543    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:16:47.884483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.877634    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:16:47.884483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.878828    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:16:47.916625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.890197    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.916625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.890518    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:47.955425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.926716    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:47.955425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:47.931514    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:16:48.016536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.011861    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:48.299754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.299646    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c\" (UniqueName: \"kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c\") pod \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") "
Oct 31 00:16:48.300668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.300632    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca\") pod \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") "
Oct 31 00:16:48.308758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.308732    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca\") pod \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\" (UID: \"656b98cc-a8be-402e-ac4e-a90ec2f12c38\") "
Oct 31 00:16:48.309709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.304199    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:16:48.368427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.344628    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:16:48.424759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.395353    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:48.463816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.463786    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:48.464217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.464200    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:16:48.518625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.518596    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:48.539647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.500925    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca" pod "656b98cc-a8be-402e-ac4e-a90ec2f12c38" (UID: "656b98cc-a8be-402e-ac4e-a90ec2f12c38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:48.704212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.606445    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:48.739651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.739617    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:48.749451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.621545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:48.775754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.775714    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:48.777221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.777192    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:16:48.808406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.808350    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:16:48.935744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.811993    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:48.935744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.885541    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:48.935744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.917539    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:16:48.948543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.936924    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:48.963488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:48.957273    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:49.041301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.032638    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca" pod "656b98cc-a8be-402e-ac4e-a90ec2f12c38" (UID: "656b98cc-a8be-402e-ac4e-a90ec2f12c38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:16:49.258146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.258108    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656b98cc-a8be-402e-ac4e-a90ec2f12c38-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:49.297494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.286615    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c" pod "656b98cc-a8be-402e-ac4e-a90ec2f12c38" (UID: "656b98cc-a8be-402e-ac4e-a90ec2f12c38"). InnerVolumeSpecName "kube-api-access-9wg2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:49.350295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.350252    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:16:49.363598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.363568    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:16:49.374348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.374203    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:16:49.395479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.395429    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:16:49.437828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.437687    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:49.449524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.438770    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wg2c\" (UniqueName: \"kubernetes.io/projected/656b98cc-a8be-402e-ac4e-a90ec2f12c38-kube-api-access-9wg2c\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:49.504653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.504557    5078 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:49.549363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.549333    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"
Oct 31 00:16:49.550132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.550097    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m" event={"ID":"656b98cc-a8be-402e-ac4e-a90ec2f12c38","Type":"ContainerDied","Data":"651bc02ec3f6774c6b1c31a73b8d981ee4b8edccbe30f00d2d3ef423b4d4a564"}
Oct 31 00:16:49.550297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.550281    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="651bc02ec3f6774c6b1c31a73b8d981ee4b8edccbe30f00d2d3ef423b4d4a564"
Oct 31 00:16:49.601920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.601889    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:16:49.603500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.603480    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:16:49.629858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.625645    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"]
Oct 31 00:16:49.629858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.626670    5078 cpu_manager.go:414] "containerMap: removing container" podUID="656b98cc-a8be-402e-ac4e-a90ec2f12c38" containerName="revoke-keys"
Oct 31 00:16:49.629858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.626690    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656b98cc-a8be-402e-ac4e-a90ec2f12c38" containerName="revoke-keys"
Oct 31 00:16:49.629858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.626896    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="656b98cc-a8be-402e-ac4e-a90ec2f12c38" containerName="revoke-keys"
Oct 31 00:16:49.640723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.640694    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"]
Oct 31 00:16:49.641031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.641011    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.674247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.674222    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:16:49.708106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.708079    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:16:49.734586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.734562    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"]
Oct 31 00:16:49.751607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.749932    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:49.759480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.758845    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.782807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.782777    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"]
Oct 31 00:16:49.788664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.788639    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.790027    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.813032    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.803064    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.813932    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tknn\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.813965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.814014    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.814052    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.814165    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.814262    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.814348    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.816486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.814383    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.820071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.819234    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:16:49.859715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.859688    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:16:49.861230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.861205    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"]
Oct 31 00:16:49.872215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.872156    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"]
Oct 31 00:16:49.883553 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:16:49.883524    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656b98cc_a8be_402e_ac4e_a90ec2f12c38.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:16:49.886716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.886694    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"]
Oct 31 00:16:49.903518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.903491    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"]
Oct 31 00:16:49.903675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.903573    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:49.910268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.910244    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:16:49.913115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.913093    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:16:49.915401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915359    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tknn\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.915509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915407    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.915509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915448    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.915582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915534    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.915616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915579    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.915650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915625    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.915700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915669    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.915700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915693    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.915806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915789    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.915882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915837    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.915882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.915876    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgmg\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.917047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.917006    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.935708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.933931    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.936693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.936282    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79z6\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.939753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.939717    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.940251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.940234    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.940419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.940404    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.940645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.940629    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.941137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941113    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.941277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941262    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.941389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941379    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.941475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941448    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.941549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941534    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.941607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941584    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.941664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941406    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.941664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941650    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.941766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941697    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:49.941815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941768    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.941873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941823    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.941873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.941856    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:49.942149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.942124    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.942379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.942356    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.942722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.942697    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.944992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.944937    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tknn\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.961079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.960342    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:49.963153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:49.963130    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:50.016821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.016539    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044212    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044244    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79z6\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044287    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044312    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044332    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044354    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044399    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044424    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044455    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.044535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044512    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044550    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044620    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044647    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044713    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044821    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5mz\" (UniqueName: \"kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044864    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044893    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.044918    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.045001    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.045063    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.045118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.045107    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgmg\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.046378    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.049497    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.049942    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.050334    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.050550    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.050719    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.050867    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.051188    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.052275    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.054534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.052370    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.063110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.062329    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.063110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.062742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.063433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.063386    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.069053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.068985    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.133453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.133383    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.133679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.133448    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgmg\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.135264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.135235    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.152937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.152850    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79z6\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.153183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.153158    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5mz\" (UniqueName: \"kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.153260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.153230    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:50.153260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.153253    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.156876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.153445    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.156876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.155204    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.160628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.158785    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.194777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.194300    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5mz\" (UniqueName: \"kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz\") pod \"maas-api-revoke-sub-8426e0fd-8pdsx\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.205503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.205444    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:50.244576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.244548    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:50.408229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.408183    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"]
Oct 31 00:16:50.409404 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:50.409367    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81265e6e_0c63_443e_9886_a53e9307af65.slice/crio-3e9a34039312eb7db7c2bdd1ec18dcd4230b11af20cba8400a2db99712f29c4d WatchSource:0}: Error finding container 3e9a34039312eb7db7c2bdd1ec18dcd4230b11af20cba8400a2db99712f29c4d: Status 404 returned error can't find the container with id 3e9a34039312eb7db7c2bdd1ec18dcd4230b11af20cba8400a2db99712f29c4d
Oct 31 00:16:50.413534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.413489    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.413634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.413582    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.413634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.413628    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.563444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.562068    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" event={"ID":"81265e6e-0c63-443e-9886-a53e9307af65","Type":"ContainerStarted","Data":"177dc1749989201340a07dbf9f171c0ba08ec26081e87ece26acc3a00831710b"}
Oct 31 00:16:50.563444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.562149    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" event={"ID":"81265e6e-0c63-443e-9886-a53e9307af65","Type":"ContainerStarted","Data":"3e9a34039312eb7db7c2bdd1ec18dcd4230b11af20cba8400a2db99712f29c4d"}
Oct 31 00:16:50.591567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.590048    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podStartSLOduration=3.590027448 podStartE2EDuration="3.590027448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:50.58872378 +0000 UTC m=+1378.535094501" watchObservedRunningTime="2026-10-31 00:16:50.590027448 +0000 UTC m=+1378.536398166"
Oct 31 00:16:50.605132 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:50.605083    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0fcca2_d2a7_4b96_8271_151d57fa2587.slice/crio-5f56eb583ec9eed418c85b3872e0d2901ba2b22846eb476850bbf591319c40a0 WatchSource:0}: Error finding container 5f56eb583ec9eed418c85b3872e0d2901ba2b22846eb476850bbf591319c40a0: Status 404 returned error can't find the container with id 5f56eb583ec9eed418c85b3872e0d2901ba2b22846eb476850bbf591319c40a0
Oct 31 00:16:50.613878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.613844    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.614070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.614051    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.614201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.614186    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.635104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.635060    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"]
Oct 31 00:16:50.701818 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:50.701762    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28464d3c_03c3_4201_b085_2d9406b99c36.slice/crio-c34e8be38b16f376e262222a575c8d5b9d6cbf8c37ecb8baf3ef52e07c6d6305 WatchSource:0}: Error finding container c34e8be38b16f376e262222a575c8d5b9d6cbf8c37ecb8baf3ef52e07c6d6305: Status 404 returned error can't find the container with id c34e8be38b16f376e262222a575c8d5b9d6cbf8c37ecb8baf3ef52e07c6d6305
Oct 31 00:16:50.708599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.708572    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"]
Oct 31 00:16:50.724618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.724588    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"]
Oct 31 00:16:50.733723 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:16:50.733670    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905c8138_3bf0_41f0_b1db_fa377aa449c3.slice/crio-e59aae555a9e2460d5024d4b51b89e7b885ef8aa7c627bcb7b33fd80cc7a8515 WatchSource:0}: Error finding container e59aae555a9e2460d5024d4b51b89e7b885ef8aa7c627bcb7b33fd80cc7a8515: Status 404 returned error can't find the container with id e59aae555a9e2460d5024d4b51b89e7b885ef8aa7c627bcb7b33fd80cc7a8515
Oct 31 00:16:50.751679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.749877    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.751679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.749976    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:50.751679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:50.750018    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:16:51.021050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.020952    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:51.025230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.025207    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:51.025735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.025412    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:51.601713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.592822    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" event={"ID":"3d0fcca2-d2a7-4b96-8271-151d57fa2587","Type":"ContainerStarted","Data":"6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92"}
Oct 31 00:16:51.601713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.592891    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" event={"ID":"3d0fcca2-d2a7-4b96-8271-151d57fa2587","Type":"ContainerStarted","Data":"5f56eb583ec9eed418c85b3872e0d2901ba2b22846eb476850bbf591319c40a0"}
Oct 31 00:16:51.621818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.605298    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx" event={"ID":"28464d3c-03c3-4201-b085-2d9406b99c36","Type":"ContainerStarted","Data":"93fd29d880abd6e675c1f72cb4419f8265314c7ae689a18ef2aa22df2413f7db"}
Oct 31 00:16:51.621818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.605363    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx" event={"ID":"28464d3c-03c3-4201-b085-2d9406b99c36","Type":"ContainerStarted","Data":"c34e8be38b16f376e262222a575c8d5b9d6cbf8c37ecb8baf3ef52e07c6d6305"}
Oct 31 00:16:51.651098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.651053    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" event={"ID":"905c8138-3bf0-41f0-b1db-fa377aa449c3","Type":"ContainerStarted","Data":"c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c"}
Oct 31 00:16:51.651279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.651127    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" event={"ID":"905c8138-3bf0-41f0-b1db-fa377aa449c3","Type":"ContainerStarted","Data":"e59aae555a9e2460d5024d4b51b89e7b885ef8aa7c627bcb7b33fd80cc7a8515"}
Oct 31 00:16:51.697488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.696832    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podStartSLOduration=3.696808989 podStartE2EDuration="3.696808989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:51.645777707 +0000 UTC m=+1379.592148499" watchObservedRunningTime="2026-10-31 00:16:51.696808989 +0000 UTC m=+1379.643179707"
Oct 31 00:16:51.697488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.697025    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx" podStartSLOduration=3.697013986 podStartE2EDuration="3.697013986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:51.668447622 +0000 UTC m=+1379.614818341" watchObservedRunningTime="2026-10-31 00:16:51.697013986 +0000 UTC m=+1379.643384703"
Oct 31 00:16:51.732488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:51.723174    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podStartSLOduration=3.723157146 podStartE2EDuration="3.723157146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:51.720953808 +0000 UTC m=+1379.667324526" watchObservedRunningTime="2026-10-31 00:16:51.723157146 +0000 UTC m=+1379.669527863"
Oct 31 00:16:52.022486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.022164    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:52.022486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.022246    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:52.160487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.159619    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:16:52.172491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.169588    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:52.172491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.169652    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:52.375550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.369527    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:52.375550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.369600    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:52.535486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.527041    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:16:52.821806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.821754    5078 generic.go:356] "Generic (PLEG): container finished" podID="28464d3c-03c3-4201-b085-2d9406b99c36" containerID="93fd29d880abd6e675c1f72cb4419f8265314c7ae689a18ef2aa22df2413f7db" exitCode=0
Oct 31 00:16:52.831500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.827612    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx" event={"ID":"28464d3c-03c3-4201-b085-2d9406b99c36","Type":"ContainerDied","Data":"93fd29d880abd6e675c1f72cb4419f8265314c7ae689a18ef2aa22df2413f7db"}
Oct 31 00:16:52.916072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:52.916029    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:53.052396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.052363    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:53.054639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.054600    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:53.154692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.154659    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:53.156718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.156669    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:53.236682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.236647    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:53.237067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.237031    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:53.536794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.536691    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:53.778487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.775558    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:16:53.789703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:53.789603    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:16:54.022283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.020918    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:54.022283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.020981    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:54.167488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.160987    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:54.167488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.161049    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:54.243515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.226628    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:54.243515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.226690    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:54.562543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.561551    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:54.773489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.741002    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:54.773489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.741338    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:16:54.773489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.743353    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:54.808488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.800574    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:54.808488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.801023    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:54.808488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.801091    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:54.808488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:54.801404    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:16:55.078252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.078212    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:55.153826 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.095670    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:16:55.165517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.162578    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:55.165517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.162632    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:55.214728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.214153    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:55.214728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.214207    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:55.216293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.216262    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:55.351564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.331132    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:55.355485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.352810    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:16:55.513488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.513012    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz\" (UniqueName: \"kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz\") pod \"28464d3c-03c3-4201-b085-2d9406b99c36\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") "
Oct 31 00:16:55.558487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.557240    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz" pod "28464d3c-03c3-4201-b085-2d9406b99c36" (UID: "28464d3c-03c3-4201-b085-2d9406b99c36"). InnerVolumeSpecName "kube-api-access-xn5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:55.627562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.619362    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca\") pod \"28464d3c-03c3-4201-b085-2d9406b99c36\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") "
Oct 31 00:16:55.627562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.619448    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca\") pod \"28464d3c-03c3-4201-b085-2d9406b99c36\" (UID: \"28464d3c-03c3-4201-b085-2d9406b99c36\") "
Oct 31 00:16:55.627562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.620411    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5mz\" (UniqueName: \"kubernetes.io/projected/28464d3c-03c3-4201-b085-2d9406b99c36-kube-api-access-xn5mz\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:55.646513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.643692    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca" pod "28464d3c-03c3-4201-b085-2d9406b99c36" (UID: "28464d3c-03c3-4201-b085-2d9406b99c36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:55.738491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.736944    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:55.767489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.766558    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca" pod "28464d3c-03c3-4201-b085-2d9406b99c36" (UID: "28464d3c-03c3-4201-b085-2d9406b99c36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:16:55.767489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.766672    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:55.767489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.767035    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:16:55.767489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.767300    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:55.767857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.767561    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:16:55.775485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.768134    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:16:55.775485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.768192    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:16:55.775485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.768225    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:16:55.795488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.779579    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:16:55.844490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:55.841004    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:16:56.069497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.056627    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:16:56.094488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.092881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:16:56.197496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.196353    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:56.340544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.322803    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28464d3c-03c3-4201-b085-2d9406b99c36-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:56.422487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.403913    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:56.422487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.403989    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:56.466329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.462700    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:56.466329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.462784    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:56.499489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.499140    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:56.772353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.771598    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:56.870427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.844224    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:16:56.944489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.919509    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 31 00:16:57.081546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.890710    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:16:57.081546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.070179    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:57.081546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.991112    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:57.081546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.076907    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:57.088485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:56.967797    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:16:57.093497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.093290    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:57.121486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.093874    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:16:57.121486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.107794    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:16:57.122335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.122292    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:57.122567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.122541    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:57.203121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.203091    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:57.203397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.203352    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:57.279490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.254637    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:57.279836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.279801    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:57.360667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.352448    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx" event={"ID":"28464d3c-03c3-4201-b085-2d9406b99c36","Type":"ContainerDied","Data":"c34e8be38b16f376e262222a575c8d5b9d6cbf8c37ecb8baf3ef52e07c6d6305"}
Oct 31 00:16:57.360955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.360934    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34e8be38b16f376e262222a575c8d5b9d6cbf8c37ecb8baf3ef52e07c6d6305"
Oct 31 00:16:57.361063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.352590    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"
Oct 31 00:16:57.532486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.508934    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:57.555080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.552221    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:57.636546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.626668    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:16:57.737494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.683144    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:16:57.863524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.856813    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 31 00:16:57.887721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.887626    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:16:57.931854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.927921    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:16:57.931854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.930610    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:16:57.980445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.952255    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:16:57.980445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.952314    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:16:57.980445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:57.952350    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:16:58.058125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.057729    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 00:16:58.058125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.057797    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 00:16:58.058125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.057956    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:16:58.154734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.154560    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:58.154734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.154624    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:58.206608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.206550    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:58.216614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.216553    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:58.556132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.552869    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:58.579314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.579279    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"]
Oct 31 00:16:58.581354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.581328    5078 cpu_manager.go:414] "containerMap: removing container" podUID="28464d3c-03c3-4201-b085-2d9406b99c36" containerName="revoke-keys"
Oct 31 00:16:58.581554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.581538    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28464d3c-03c3-4201-b085-2d9406b99c36" containerName="revoke-keys"
Oct 31 00:16:58.582233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.582175    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="28464d3c-03c3-4201-b085-2d9406b99c36" containerName="revoke-keys"
Oct 31 00:16:58.615486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.593993    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.615642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.615554    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.615754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.615727    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.620489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.615850    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4h7p\" (UniqueName: \"kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.630488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.628538    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"]
Oct 31 00:16:58.637487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.637046    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:16:58.646484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.643045    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:16:58.646484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.644885    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:16:58.731489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.719994    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.737188 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.737153    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4h7p\" (UniqueName: \"kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.737547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.724634    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.737781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.737759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.743835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.743807    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:58.796206 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:58.795906    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4h7p\" (UniqueName: \"kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p\") pod \"maas-api-revoke-sub-dbf200d6-k2prv\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:59.045116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.040781    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:16:59.118608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.118577    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:59.120299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.120271    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:59.127481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.126726    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:16:59.157769 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.156832    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 00:16:59.157769 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.156888    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 00:16:59.220484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.206868    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:16:59.220484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.206921    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 31 00:16:59.457235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:16:59.457207    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"]
Oct 31 00:17:00.225492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.222348    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:17:00.225492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.223195    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:17:00.225492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.224300    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:17:00.244519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.244488    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:17:00.250028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.249998    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:17:00.252386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.252365    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:17:00.595542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.594840    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv" event={"ID":"c9d08871-b160-46b6-a606-91638cdfcd52","Type":"ContainerStarted","Data":"f1375076c0cef7628890747b1ec1fa485d8748955209cf368d0a47838fb08453"}
Oct 31 00:17:00.595542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.594948    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv" event={"ID":"c9d08871-b160-46b6-a606-91638cdfcd52","Type":"ContainerStarted","Data":"edf15f5d74157094a2039dcbb2c907f278094b3cf42c8605725bc10b2752cb0b"}
Oct 31 00:17:00.619494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:00.616297    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv" podStartSLOduration=2.616272507 podStartE2EDuration="2.616272507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:00.614875584 +0000 UTC m=+1388.561246302" watchObservedRunningTime="2026-10-31 00:17:00.616272507 +0000 UTC m=+1388.562643223"
Oct 31 00:17:01.605420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:01.605323    5078 generic.go:356] "Generic (PLEG): container finished" podID="c9d08871-b160-46b6-a606-91638cdfcd52" containerID="f1375076c0cef7628890747b1ec1fa485d8748955209cf368d0a47838fb08453" exitCode=0
Oct 31 00:17:01.605887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:01.605426    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv" event={"ID":"c9d08871-b160-46b6-a606-91638cdfcd52","Type":"ContainerDied","Data":"f1375076c0cef7628890747b1ec1fa485d8748955209cf368d0a47838fb08453"}
Oct 31 00:17:02.638211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.638171    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"]
Oct 31 00:17:02.643412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.643384    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.663272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.663241    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"]
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.721909    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722043    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722076    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722160    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722195    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffl2p\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722241    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722291    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722344    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.722738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.722435    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836204    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836344    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836373    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836450    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836507    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffl2p\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836554    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836605    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836657    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.836821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.836780    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.838125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.837420    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.838125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.837421    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.838125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.837689    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.838125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.837928    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.838125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.838086    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.840488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.840437    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.840853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.840807    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.854516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.853799    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.858956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.858928    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffl2p\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:02.892966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.892939    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:17:02.961280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:02.961237    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:03.041815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.040477    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca\") pod \"c9d08871-b160-46b6-a606-91638cdfcd52\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") "
Oct 31 00:17:03.041815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.040718    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca\") pod \"c9d08871-b160-46b6-a606-91638cdfcd52\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") "
Oct 31 00:17:03.041815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.040763    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p\" (UniqueName: \"kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p\") pod \"c9d08871-b160-46b6-a606-91638cdfcd52\" (UID: \"c9d08871-b160-46b6-a606-91638cdfcd52\") "
Oct 31 00:17:03.043757 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.043709    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca" pod "c9d08871-b160-46b6-a606-91638cdfcd52" (UID: "c9d08871-b160-46b6-a606-91638cdfcd52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:03.046230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.045281    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p" pod "c9d08871-b160-46b6-a606-91638cdfcd52" (UID: "c9d08871-b160-46b6-a606-91638cdfcd52"). InnerVolumeSpecName "kube-api-access-j4h7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:03.054138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.050987    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca" pod "c9d08871-b160-46b6-a606-91638cdfcd52" (UID: "c9d08871-b160-46b6-a606-91638cdfcd52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:03.085976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.085946    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"]
Oct 31 00:17:03.088104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.086970    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c9d08871-b160-46b6-a606-91638cdfcd52" containerName="revoke-keys"
Oct 31 00:17:03.088104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.086992    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d08871-b160-46b6-a606-91638cdfcd52" containerName="revoke-keys"
Oct 31 00:17:03.088104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.087197    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d08871-b160-46b6-a606-91638cdfcd52" containerName="revoke-keys"
Oct 31 00:17:03.130540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.126410    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"]
Oct 31 00:17:03.130540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.126582    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.154765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.147912    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:03.154765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.147947    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4h7p\" (UniqueName: \"kubernetes.io/projected/c9d08871-b160-46b6-a606-91638cdfcd52-kube-api-access-j4h7p\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:03.154765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.147970    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d08871-b160-46b6-a606-91638cdfcd52-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:03.249416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.249385    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.249582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.249455    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.249682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.249661    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2khmt\" (UniqueName: \"kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.259370 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:03.259342    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd398e49a_c635_4945_8971_067877a21aea.slice/crio-0d05991466c326f40558784f2fc6f6248e80cc7b35b59100e73c7714cffc6475 WatchSource:0}: Error finding container 0d05991466c326f40558784f2fc6f6248e80cc7b35b59100e73c7714cffc6475: Status 404 returned error can't find the container with id 0d05991466c326f40558784f2fc6f6248e80cc7b35b59100e73c7714cffc6475
Oct 31 00:17:03.260655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.260632    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"]
Oct 31 00:17:03.261950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.261919    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:03.262006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.261981    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:03.262044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.262024    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:03.351787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.351751    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.352306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.352275    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.352629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.352605    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2khmt\" (UniqueName: \"kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.353441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.353417    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.354825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.354800    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.366574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.366546    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2khmt\" (UniqueName: \"kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt\") pod \"maas-api-revoke-sub-26fb7049-69vkv\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.467267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.467187    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:03.634441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.633869    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" event={"ID":"d398e49a-c635-4945-8971-067877a21aea","Type":"ContainerStarted","Data":"947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e"}
Oct 31 00:17:03.634441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.633944    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" event={"ID":"d398e49a-c635-4945-8971-067877a21aea","Type":"ContainerStarted","Data":"0d05991466c326f40558784f2fc6f6248e80cc7b35b59100e73c7714cffc6475"}
Oct 31 00:17:03.635852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.635818    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv" event={"ID":"c9d08871-b160-46b6-a606-91638cdfcd52","Type":"ContainerDied","Data":"edf15f5d74157094a2039dcbb2c907f278094b3cf42c8605725bc10b2752cb0b"}
Oct 31 00:17:03.635960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.635871    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf15f5d74157094a2039dcbb2c907f278094b3cf42c8605725bc10b2752cb0b"
Oct 31 00:17:03.635960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.635951    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"
Oct 31 00:17:03.663134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.663088    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podStartSLOduration=1.663074606 podStartE2EDuration="1.663074606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:03.659587707 +0000 UTC m=+1391.605958424" watchObservedRunningTime="2026-10-31 00:17:03.663074606 +0000 UTC m=+1391.609445323"
Oct 31 00:17:03.775026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.774999    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"]
Oct 31 00:17:03.961798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:03.961769    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:04.647762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:04.647703    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv" event={"ID":"70cd632a-5414-465b-9406-b94e59152783","Type":"ContainerStarted","Data":"fdd1f30822e3381ed10c187ada98072a3c6c51ae970476f982da2a5969064b05"}
Oct 31 00:17:04.648430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:04.648394    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv" event={"ID":"70cd632a-5414-465b-9406-b94e59152783","Type":"ContainerStarted","Data":"426fdce1db3732d7d1dcbde2f63ba80b7cee8449c88d13ebf8c281830903feb5"}
Oct 31 00:17:04.695490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:04.691800    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv" podStartSLOduration=1.691776005 podStartE2EDuration="1.691776005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:04.667522597 +0000 UTC m=+1392.613893316" watchObservedRunningTime="2026-10-31 00:17:04.691776005 +0000 UTC m=+1392.638146725"
Oct 31 00:17:04.963362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:04.963273    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:17:04.963362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:04.963328    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 00:17:05.667736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:05.667698    5078 generic.go:356] "Generic (PLEG): container finished" podID="70cd632a-5414-465b-9406-b94e59152783" containerID="fdd1f30822e3381ed10c187ada98072a3c6c51ae970476f982da2a5969064b05" exitCode=0
Oct 31 00:17:05.667909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:05.667784    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv" event={"ID":"70cd632a-5414-465b-9406-b94e59152783","Type":"ContainerDied","Data":"fdd1f30822e3381ed10c187ada98072a3c6c51ae970476f982da2a5969064b05"}
Oct 31 00:17:05.961833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:05.961750    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:17:05.961833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:05.961805    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 00:17:06.411400 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:17:06.411359    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:17:06.956020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:06.955999    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:06.962012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:06.961959    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:17:06.962012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:06.962002    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 00:17:07.053361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.053327    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca\") pod \"70cd632a-5414-465b-9406-b94e59152783\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") "
Oct 31 00:17:07.053580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.053566    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca\") pod \"70cd632a-5414-465b-9406-b94e59152783\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") "
Oct 31 00:17:07.053696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.053592    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt\" (UniqueName: \"kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt\") pod \"70cd632a-5414-465b-9406-b94e59152783\" (UID: \"70cd632a-5414-465b-9406-b94e59152783\") "
Oct 31 00:17:07.053916 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.053889    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca" pod "70cd632a-5414-465b-9406-b94e59152783" (UID: "70cd632a-5414-465b-9406-b94e59152783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:07.054089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.054068    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70cd632a-5414-465b-9406-b94e59152783-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:07.055741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.055719    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca" pod "70cd632a-5414-465b-9406-b94e59152783" (UID: "70cd632a-5414-465b-9406-b94e59152783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:07.055887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.055866    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt" pod "70cd632a-5414-465b-9406-b94e59152783" (UID: "70cd632a-5414-465b-9406-b94e59152783"). InnerVolumeSpecName "kube-api-access-2khmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:07.155685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.155650    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2khmt\" (UniqueName: \"kubernetes.io/projected/70cd632a-5414-465b-9406-b94e59152783-kube-api-access-2khmt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:07.155685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.155685    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70cd632a-5414-465b-9406-b94e59152783-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:07.723515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.723391    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv" event={"ID":"70cd632a-5414-465b-9406-b94e59152783","Type":"ContainerDied","Data":"426fdce1db3732d7d1dcbde2f63ba80b7cee8449c88d13ebf8c281830903feb5"}
Oct 31 00:17:07.723515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.723455    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426fdce1db3732d7d1dcbde2f63ba80b7cee8449c88d13ebf8c281830903feb5"
Oct 31 00:17:07.723797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.723572    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"
Oct 31 00:17:07.962836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.962809    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:17:07.963346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:07.963315    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 00:17:08.961606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:08.961526    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:17:08.961606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:08.961578    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 00:17:09.968889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:09.968853    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:17:09.969515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:09.969454    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 00:17:10.690488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:10.688501    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:10.811489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:10.809028    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:10.811489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:10.810143    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:17:10.980544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:10.977561    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.046630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.045632    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:11.082607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.082558    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:11.121639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.121586    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.157675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.157620    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:11.366664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.366554    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.492653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.492602    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.580491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.568828    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.614645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.614598    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.696646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.696596    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.697810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.697785    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:17:11.813156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.813111    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:11.817487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:11.814036    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:17:12.055410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.055295    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.088629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.088572    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.139493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.130211    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:17:12.139493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.130282    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:17:12.204488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.198148    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.265607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.265567    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:12.266978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.266896    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:12.417177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.404146    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:17:12.417177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.414223    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.518635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.518594    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:12.536333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.536291    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.579707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.575655    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.644833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.644792    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:12.720847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.719547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.747874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.747813    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:12.793086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.793046    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.798744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.798702    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:12.831532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.809101    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:12.831532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.809173    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.831924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.802538    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.832126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.832099    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.842504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.793553    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:12.846484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.842843    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:12.851547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.846666    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:12.956000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.955972    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:17:12.985291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.985206    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:17:12.985890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.985873    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:17:12.986207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.986194    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:17:12.986867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.986844    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:17:12.987194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:12.987181    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:17:13.083779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.081708    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.112580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.105327    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.310578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.210673    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:13.310578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.304648    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.310578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.307253    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.435489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.434722    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.558490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.557897    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:13.583540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.581424    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:13.637489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.611127    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:17:13.662617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.642571    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.694499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.675090    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.723332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.723115    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:13.797621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:13.767566    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.042008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.023631    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.042008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.024070    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:17:14.042008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.040614    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 31 00:17:14.042570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.042251    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.149349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.112410    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:14.303369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.303275    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:14.303673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.303639    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.303999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.303974    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.305260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.305234    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:14.307776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.307132    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:14.315714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.312592    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.474796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.450562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.598110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.596823    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.688338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.688286    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.737585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.715255    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.737585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.727843    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.737585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.734066    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.759342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.759309    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:14.759856 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.759823    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.774911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.774870    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:14.964400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.964362    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:14.964767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.964731    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:14.968531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:14.968424    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:17:15.042484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.029971    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:15.043692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.043655    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:17:15.115871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.115842    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:17:15.120573 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.120540    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:15.137771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.137737    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:15.149038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.149010    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:17:15.240800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.240707    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:17:15.331496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.330791    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:17:15.331496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.330871    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:17:15.332243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.314849    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:15.341911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.341887    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:17:15.370807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.365148    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:17:15.370807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.365231    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:17:15.370807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.365281    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:17:15.385869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.385828    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"]
Oct 31 00:17:15.387186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.387145    5078 cpu_manager.go:414] "containerMap: removing container" podUID="70cd632a-5414-465b-9406-b94e59152783" containerName="revoke-keys"
Oct 31 00:17:15.387353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.387340    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cd632a-5414-465b-9406-b94e59152783" containerName="revoke-keys"
Oct 31 00:17:15.387643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.387627    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cd632a-5414-465b-9406-b94e59152783" containerName="revoke-keys"
Oct 31 00:17:15.393067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.393049    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"]
Oct 31 00:17:15.393502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.393472    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.401577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.401030    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:17:15.403417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.403394    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:17:15.404505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.404393    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:17:15.404631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.404581    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:15.578937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.578832    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tmbf\" (UniqueName: \"kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.579109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.579071    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.579174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.579133    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.681004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.680601    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tmbf\" (UniqueName: \"kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.681510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.681483    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.682517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.682497    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.683133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.682372    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.694241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.694173    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.694645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.694623    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tmbf\" (UniqueName: \"kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf\") pod \"maas-api-revoke-sub-d79769d5-nfjkq\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:15.738408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:15.738130    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:16.047915 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:16.047880    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5e64c1_0326_4a0b_97c6_87ff734a6521.slice/crio-2bee2c59b30dc2b21a73a4346ace6436c8f84ce9a348f09030c58f314b70d9ed WatchSource:0}: Error finding container 2bee2c59b30dc2b21a73a4346ace6436c8f84ce9a348f09030c58f314b70d9ed: Status 404 returned error can't find the container with id 2bee2c59b30dc2b21a73a4346ace6436c8f84ce9a348f09030c58f314b70d9ed
Oct 31 00:17:16.049031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:16.049007    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"]
Oct 31 00:17:16.079139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:16.079058    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq" event={"ID":"6c5e64c1-0326-4a0b-97c6-87ff734a6521","Type":"ContainerStarted","Data":"2bee2c59b30dc2b21a73a4346ace6436c8f84ce9a348f09030c58f314b70d9ed"}
Oct 31 00:17:17.090700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.090668    5078 generic.go:356] "Generic (PLEG): container finished" podID="6c5e64c1-0326-4a0b-97c6-87ff734a6521" containerID="11d7f77fce6632eeb30dd9cd9b2ff6a5ac3c873d7382a0045e223c18a626f0cf" exitCode=0
Oct 31 00:17:17.091025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.090743    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq" event={"ID":"6c5e64c1-0326-4a0b-97c6-87ff734a6521","Type":"ContainerDied","Data":"11d7f77fce6632eeb30dd9cd9b2ff6a5ac3c873d7382a0045e223c18a626f0cf"}
Oct 31 00:17:17.813866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.813832    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"]
Oct 31 00:17:17.819365 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.819341    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.840211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.840184    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"]
Oct 31 00:17:17.949327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949288    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949356    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949425    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949584    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5w6\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949614    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949705    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949756    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.949968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949943    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.950085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.949994    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:17.998860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:17.998813    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"]
Oct 31 00:17:18.024717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.024673    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"]
Oct 31 00:17:18.024940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.024808    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.051546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051511    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.051708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051590    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.051786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051750    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.051851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051794    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.051910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051871    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.051967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051921    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.051967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.051951    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.052100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.052080    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.052215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.052197    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5w6\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.052277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.052232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.052694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.052669    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.052904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.052882    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.053373 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.053351    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.053428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.053352    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.056021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.055996    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.056357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.056309    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.079278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.079210    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.085586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.085536    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5w6\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.145070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.144971    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:18.153776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.153745    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.153890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.153810    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.153949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.153918    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qm8\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.153949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.153945    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.154054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.154040    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.154114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.154055    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.154114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.154091    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.154219 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.154177    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.154270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.154236    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261486    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261582    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qm8\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261625    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261730    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261758    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261861    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.261949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.261927    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.262039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.262019    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.262158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.262142    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.262230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.262220    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.263767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.263175    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.263767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.263376    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.263767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.263494    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.263953 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.263916    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.266530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.266507    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.266617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.266539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.287590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.287563    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qm8\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.290080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.290053    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.296067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.296048    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:18.347979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.347716    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:18.351262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.351238    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"]
Oct 31 00:17:18.352136 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.352112    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6c5e64c1-0326-4a0b-97c6-87ff734a6521" containerName="revoke-keys"
Oct 31 00:17:18.352220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.352138    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5e64c1-0326-4a0b-97c6-87ff734a6521" containerName="revoke-keys"
Oct 31 00:17:18.352370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.352349    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5e64c1-0326-4a0b-97c6-87ff734a6521" containerName="revoke-keys"
Oct 31 00:17:18.359827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.359806    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.367832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.367542    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca\") pod \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") "
Oct 31 00:17:18.367832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.367662    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf\" (UniqueName: \"kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf\") pod \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") "
Oct 31 00:17:18.368236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.368209    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca\") pod \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\" (UID: \"6c5e64c1-0326-4a0b-97c6-87ff734a6521\") "
Oct 31 00:17:18.369295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.369259    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca" pod "6c5e64c1-0326-4a0b-97c6-87ff734a6521" (UID: "6c5e64c1-0326-4a0b-97c6-87ff734a6521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:18.372196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.372064    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca" pod "6c5e64c1-0326-4a0b-97c6-87ff734a6521" (UID: "6c5e64c1-0326-4a0b-97c6-87ff734a6521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:18.383342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.383311    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"]
Oct 31 00:17:18.386704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.386675    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf" pod "6c5e64c1-0326-4a0b-97c6-87ff734a6521" (UID: "6c5e64c1-0326-4a0b-97c6-87ff734a6521"). InnerVolumeSpecName "kube-api-access-2tmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:18.473549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.473511    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.473896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.473875    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7dm\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.474123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.474106    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.474299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.474284    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.474597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.474581    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.474844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.474828    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.475012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.474998    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.475201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.475187    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.475451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.475435    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.475693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.475679    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:18.475798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.475786    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tmbf\" (UniqueName: \"kubernetes.io/projected/6c5e64c1-0326-4a0b-97c6-87ff734a6521-kube-api-access-2tmbf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:18.475887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.475877    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e64c1-0326-4a0b-97c6-87ff734a6521-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:18.476189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.476170    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"]
Oct 31 00:17:18.482197 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:18.482151    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d43026d_495b_4d5c_b30c_4ee2f68bb8f4.slice/crio-7f69cb793cbdfb7bdd80840dbca90f5665f20550233a54aeb242facc3323e7be WatchSource:0}: Error finding container 7f69cb793cbdfb7bdd80840dbca90f5665f20550233a54aeb242facc3323e7be: Status 404 returned error can't find the container with id 7f69cb793cbdfb7bdd80840dbca90f5665f20550233a54aeb242facc3323e7be
Oct 31 00:17:18.485245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.485212    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:18.485377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.485325    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:18.485440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.485374    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.577955    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578085    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7dm\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578151    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578189    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578328    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578358    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578415    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578531    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.578989    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.579603    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.580498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.580144    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.581229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.580883    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.581229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.580906    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.584914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.584886    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.586793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.586769    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.596707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.596679    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7dm\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.598928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.598864    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.693643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.693373    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:18.703972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.703236    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:18.703972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.703325    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:18.703972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.703366    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:18.706077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:18.706047    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"]
Oct 31 00:17:19.025834 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:19.025224    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1d12b5_4ac9_46e9_a368_36f48a923c59.slice/crio-fdb6debcd3cb808b6d889261dc764216bd0bf52ddf5408edbcd8e0d19585479d WatchSource:0}: Error finding container fdb6debcd3cb808b6d889261dc764216bd0bf52ddf5408edbcd8e0d19585479d: Status 404 returned error can't find the container with id fdb6debcd3cb808b6d889261dc764216bd0bf52ddf5408edbcd8e0d19585479d
Oct 31 00:17:19.025834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.025786    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"]
Oct 31 00:17:19.039663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.034360    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:19.039663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.034453    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:19.039663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.034511    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:19.134248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.133815    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" event={"ID":"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4","Type":"ContainerStarted","Data":"acb15657ba21ee15916d3dc028686548320ba5547ebd5fb4f1322bbbe93a264e"}
Oct 31 00:17:19.134248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.133873    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" event={"ID":"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4","Type":"ContainerStarted","Data":"7f69cb793cbdfb7bdd80840dbca90f5665f20550233a54aeb242facc3323e7be"}
Oct 31 00:17:19.136323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.136298    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"
Oct 31 00:17:19.136440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.136339    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq" event={"ID":"6c5e64c1-0326-4a0b-97c6-87ff734a6521","Type":"ContainerDied","Data":"2bee2c59b30dc2b21a73a4346ace6436c8f84ce9a348f09030c58f314b70d9ed"}
Oct 31 00:17:19.136440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.136384    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bee2c59b30dc2b21a73a4346ace6436c8f84ce9a348f09030c58f314b70d9ed"
Oct 31 00:17:19.139276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.138517    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" event={"ID":"0c1d12b5-4ac9-46e9-a368-36f48a923c59","Type":"ContainerStarted","Data":"fdb6debcd3cb808b6d889261dc764216bd0bf52ddf5408edbcd8e0d19585479d"}
Oct 31 00:17:19.142173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.142090    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" event={"ID":"75bc669f-f382-41b1-8a91-b79e0b976575","Type":"ContainerStarted","Data":"fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd"}
Oct 31 00:17:19.142173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.142141    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" event={"ID":"75bc669f-f382-41b1-8a91-b79e0b976575","Type":"ContainerStarted","Data":"117bc27024f20a763bb0bda76b4d3ac812aa9eec01a375ed98c20bae40370f0f"}
Oct 31 00:17:19.145252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.145208    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:19.146906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.146879    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:19.147006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.146932    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:19.170129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.170066    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podStartSLOduration=2.170047388 podStartE2EDuration="2.170047388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:19.162116518 +0000 UTC m=+1407.108487233" watchObservedRunningTime="2026-10-31 00:17:19.170047388 +0000 UTC m=+1407.116418123"
Oct 31 00:17:19.208894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.208812    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podStartSLOduration=2.208792404 podStartE2EDuration="2.208792404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:19.198599921 +0000 UTC m=+1407.144970639" watchObservedRunningTime="2026-10-31 00:17:19.208792404 +0000 UTC m=+1407.155163124"
Oct 31 00:17:19.348794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.348635    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:19.351057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.351017    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:19.351267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:19.351245    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:20.146137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.146092    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:20.146641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.146163    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:20.155609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.155573    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" event={"ID":"0c1d12b5-4ac9-46e9-a368-36f48a923c59","Type":"ContainerStarted","Data":"e7a4ea26dd6c40c4199c793c182c33b26766bbf6e67ee9f06e082a3d7d7db1be"}
Oct 31 00:17:20.252974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.252915    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podStartSLOduration=2.252894324 podStartE2EDuration="2.252894324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:20.216679285 +0000 UTC m=+1408.163050004" watchObservedRunningTime="2026-10-31 00:17:20.252894324 +0000 UTC m=+1408.199265033"
Oct 31 00:17:20.371500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.369411    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:20.371500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.369516    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:20.703792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.703760    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:20.704178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.704095    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:20.704328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:20.704307    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:21.170326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.170290    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:21.170960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.170926    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:21.354030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.353995    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:21.354317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.354288    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:21.700489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.699861    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:21.700489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.699925    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:21.931007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:21.930962    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:22.146641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.146607    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:22.146919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.146893    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:22.310589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.310547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:22.425542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.418440    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:22.425542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.418519    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:22.755540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.748036    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:22.890539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.852963    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:22.890539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.853026    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:22.890539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:22.873860    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:17:23.076562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.076040    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:23.076859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.076821    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:23.179864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.160028    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:23.263489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.233969    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:17:23.294249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.294211    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:23.319490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.294523    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:23.320600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.309612    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:23.377407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.377315    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:23.377707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.377675    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:23.393824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.393767    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:23.415490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.414043    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:23.415490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.414735    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:17:23.707634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.707597    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:23.707911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.707884    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:23.736133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.736093    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:23.838503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.815798    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:23.838882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.838846    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:23.949708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:23.880488    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:24.068836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.057348    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.080398 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.080345    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:24.084715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.084677    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.117500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.112404    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 31 00:17:24.152150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.113059    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:17:24.152588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.152563    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:17:24.252490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.240697    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:24.252490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.240769    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:24.252490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.241325    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:24.425549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.392953    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.426524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.426492    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:24.429498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.429441    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:24.502498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.487651    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.502498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.488019    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:24.810546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.792509    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.810546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.807572    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.844414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.844334    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:17:24.844723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.844704    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:17:24.844861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.844849    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:17:24.844982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.844971    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:17:24.845111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.845098    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:17:24.847128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.847101    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:24.849860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.848801    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:17:24.851536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.851510    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:24.852782 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.852759    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:17:24.956515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.956441    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:24.999839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:24.958360    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:17:25.045591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.030770    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:25.045591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.030837    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:25.131037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.092704    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.465302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.465261    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Oct 31 00:17:25.466020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.465997    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:25.466185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.466163    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:25.467326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.467304    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:25.467499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.467479    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:25.467703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.467685    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.495591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.482766    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.495591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.488493    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.520836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.520781    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.521834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.521539    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:25.543537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.524116    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:17:25.754537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.753669    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:25.754537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.753727    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:25.754760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.754684    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:17:25.791247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.791215    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:17:25.791920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.791535    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:17:25.814488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.808529    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.856619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.856583    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:25.933453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.933412    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:25.943037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.943009    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:17:25.944436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.944418    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:17:25.963495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:25.963447    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:17:26.040040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.039954    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:17:26.120664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.110432    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:26.160485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.159051    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:17:26.160485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.159128    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 00:17:26.160485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.160152    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:17:26.335865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.335369    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:17:26.364113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.364075    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 00:17:26.367548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.367482    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 00:17:26.695248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.695217    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:26.695735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:26.695702    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:27.477248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.475213    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:27.479637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.478962    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:27.481369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.481349    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:17:27.686013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.685987    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:27.687426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.687401    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:27.688902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.688881    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:17:27.695111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.695086    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 00:17:27.695246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:27.695135    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 00:17:28.699219 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:28.699189    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:28.699737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:28.699712    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:28.700587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:28.700568    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:17:29.664983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.664937    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"]
Oct 31 00:17:29.672229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.672204    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.672613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.672588    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"]
Oct 31 00:17:29.678847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.678820    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:29.680302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.680282    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:17:29.769623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.769583    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.770112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.769764    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.770112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.769838    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jqf\" (UniqueName: \"kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.841976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.841942    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"]
Oct 31 00:17:29.847073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.847046    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:29.854613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.854588    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"]
Oct 31 00:17:29.873654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.873623    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.874381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.874345    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.874544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.874421    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jqf\" (UniqueName: \"kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.875825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.875799    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.879060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.879041    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.891348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.891304    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jqf\" (UniqueName: \"kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf\") pod \"maas-api-revoke-sub-2d98f6f1-xglx8\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:29.976407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.976323    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jv6n\" (UniqueName: \"kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:29.976643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.976435    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:29.976643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.976485    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:29.995412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:29.995387    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:30.092335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.092306    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.092568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.092555    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.092872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.092860    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jv6n\" (UniqueName: \"kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.093818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.093801    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.097279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.097257    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.107411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.107379    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jv6n\" (UniqueName: \"kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n\") pod \"maas-api-revoke-sub-00e358a6-9hnmp\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.168010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.167977    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:30.314441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.314379    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"]
Oct 31 00:17:30.314788 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:30.314741    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dab808_935c_4922_a6a7_407540897107.slice/crio-e3a82d6632777c323decea568679d6ab2f035a87121226d4385a8ba85767cda4 WatchSource:0}: Error finding container e3a82d6632777c323decea568679d6ab2f035a87121226d4385a8ba85767cda4: Status 404 returned error can't find the container with id e3a82d6632777c323decea568679d6ab2f035a87121226d4385a8ba85767cda4
Oct 31 00:17:30.579643 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:30.579601    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ce096e_fcb0_471b_a842_382de2ba1100.slice/crio-8030959d4ea452ab21373bb2ff2d1a0197b395f18d99a0c2b2a0954a66aa1069 WatchSource:0}: Error finding container 8030959d4ea452ab21373bb2ff2d1a0197b395f18d99a0c2b2a0954a66aa1069: Status 404 returned error can't find the container with id 8030959d4ea452ab21373bb2ff2d1a0197b395f18d99a0c2b2a0954a66aa1069
Oct 31 00:17:30.600639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.600051    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"]
Oct 31 00:17:30.848191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.848093    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp" event={"ID":"e2ce096e-fcb0-471b-a842-382de2ba1100","Type":"ContainerStarted","Data":"8030959d4ea452ab21373bb2ff2d1a0197b395f18d99a0c2b2a0954a66aa1069"}
Oct 31 00:17:30.849802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:30.849774    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8" event={"ID":"e2dab808-935c-4922-a6a7-407540897107","Type":"ContainerStarted","Data":"e3a82d6632777c323decea568679d6ab2f035a87121226d4385a8ba85767cda4"}
Oct 31 00:17:31.860018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:31.859936    5078 generic.go:356] "Generic (PLEG): container finished" podID="e2ce096e-fcb0-471b-a842-382de2ba1100" containerID="c36ad0f04cdebee4c5d7c678bc33aa9d1753cf6a72a050271bcd6be3c11740af" exitCode=0
Oct 31 00:17:31.860406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:31.860023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp" event={"ID":"e2ce096e-fcb0-471b-a842-382de2ba1100","Type":"ContainerDied","Data":"c36ad0f04cdebee4c5d7c678bc33aa9d1753cf6a72a050271bcd6be3c11740af"}
Oct 31 00:17:31.861926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:31.861904    5078 generic.go:356] "Generic (PLEG): container finished" podID="e2dab808-935c-4922-a6a7-407540897107" containerID="7997537535a8ad28ec5143a91c1eb9c22bfac5c93dda297b6255e833b737132f" exitCode=0
Oct 31 00:17:31.862060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:31.861985    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8" event={"ID":"e2dab808-935c-4922-a6a7-407540897107","Type":"ContainerDied","Data":"7997537535a8ad28ec5143a91c1eb9c22bfac5c93dda297b6255e833b737132f"}
Oct 31 00:17:32.856741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.856709    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"]
Oct 31 00:17:32.862344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.862319    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.890252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.890163    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"]
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951450    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951577    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951613    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951677    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951707    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wj9\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951872    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.951938    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.952000    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:32.952349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:32.952057    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053155    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053211    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053277    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053306    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wj9\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053369    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053442    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053525    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053589    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053613    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053823    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.053921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.053898    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.054110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.054067    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.054182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.054129    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.054788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.054766    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.056398 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.056375    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.059097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.059075    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.081244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.078060    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.134535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.134499    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wj9\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.138567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.138541    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"]
Oct 31 00:17:33.150616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.150589    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.164079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.161631    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:33.164079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.163213    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"]
Oct 31 00:17:33.164803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.164782    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e2dab808-935c-4922-a6a7-407540897107" containerName="revoke-keys"
Oct 31 00:17:33.164803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.164801    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2dab808-935c-4922-a6a7-407540897107" containerName="revoke-keys"
Oct 31 00:17:33.164985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.164945    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2dab808-935c-4922-a6a7-407540897107" containerName="revoke-keys"
Oct 31 00:17:33.166448 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.166430    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:33.168872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.168854    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.170593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.170567    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"]
Oct 31 00:17:33.190167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.190121    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:33.205832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.204571    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"]
Oct 31 00:17:33.229359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.229336    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"]
Oct 31 00:17:33.232981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.230393    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e2ce096e-fcb0-471b-a842-382de2ba1100" containerName="revoke-keys"
Oct 31 00:17:33.232981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.230413    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ce096e-fcb0-471b-a842-382de2ba1100" containerName="revoke-keys"
Oct 31 00:17:33.232981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.230755    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ce096e-fcb0-471b-a842-382de2ba1100" containerName="revoke-keys"
Oct 31 00:17:33.242991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.242167    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"]
Oct 31 00:17:33.242991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.242271    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.259675    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf\" (UniqueName: \"kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf\") pod \"e2dab808-935c-4922-a6a7-407540897107\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") "
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.259807    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n\" (UniqueName: \"kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n\") pod \"e2ce096e-fcb0-471b-a842-382de2ba1100\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") "
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.259837    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca\") pod \"e2ce096e-fcb0-471b-a842-382de2ba1100\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") "
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.259908    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca\") pod \"e2ce096e-fcb0-471b-a842-382de2ba1100\" (UID: \"e2ce096e-fcb0-471b-a842-382de2ba1100\") "
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.259981    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca\") pod \"e2dab808-935c-4922-a6a7-407540897107\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") "
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.260037    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca\") pod \"e2dab808-935c-4922-a6a7-407540897107\" (UID: \"e2dab808-935c-4922-a6a7-407540897107\") "
Oct 31 00:17:33.261196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.260959    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca" pod "e2ce096e-fcb0-471b-a842-382de2ba1100" (UID: "e2ce096e-fcb0-471b-a842-382de2ba1100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:33.261756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.261726    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca" pod "e2dab808-935c-4922-a6a7-407540897107" (UID: "e2dab808-935c-4922-a6a7-407540897107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:33.263622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263574    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.263763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263646    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.263763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263684    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdcl\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.263880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263764    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.263880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263803    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.263880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263846    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.264036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263881    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.264036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263905    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.264036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.263940    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.264155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.264044    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.264155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.264107    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.264155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.264136    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.264299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.264281    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.264365 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.264340    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf672\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.264422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.264407    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.266164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.266132    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.266261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.266222    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.266627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.266604    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.266787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.266770    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:33.266830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.266794    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dab808-935c-4922-a6a7-407540897107-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:33.267998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.267928    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca" pod "e2ce096e-fcb0-471b-a842-382de2ba1100" (UID: "e2ce096e-fcb0-471b-a842-382de2ba1100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:33.268172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.268098    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n" pod "e2ce096e-fcb0-471b-a842-382de2ba1100" (UID: "e2ce096e-fcb0-471b-a842-382de2ba1100"). InnerVolumeSpecName "kube-api-access-5jv6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:33.269028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.268983    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf" pod "e2dab808-935c-4922-a6a7-407540897107" (UID: "e2dab808-935c-4922-a6a7-407540897107"). InnerVolumeSpecName "kube-api-access-q9jqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:33.270651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.270632    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca" pod "e2dab808-935c-4922-a6a7-407540897107" (UID: "e2dab808-935c-4922-a6a7-407540897107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:33.368497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368311    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.368497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368371    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf672\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.368497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368424    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368551    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368582    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368602    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368635    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368657    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368717    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368740    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.368791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368778    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368800    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368822    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdcl\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368856    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368888    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368914    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368940    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368964    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.368979    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369008    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369051    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmf9\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369080    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369115    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369155    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369180    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369197    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369276    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9jqf\" (UniqueName: \"kubernetes.io/projected/e2dab808-935c-4922-a6a7-407540897107-kube-api-access-q9jqf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369290    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jv6n\" (UniqueName: \"kubernetes.io/projected/e2ce096e-fcb0-471b-a842-382de2ba1100-kube-api-access-5jv6n\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369300    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ce096e-fcb0-471b-a842-382de2ba1100-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:33.369397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.369311    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dab808-935c-4922-a6a7-407540897107-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:33.370331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.370006    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.370836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.370784    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.371091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.371069    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.371161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.371135    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.373161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.371353    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.373161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.371609    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.373161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.371821    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.373161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.370784    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.373827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.373776    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.374674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.374490    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.374674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.374633    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.375113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.375093    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.375439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.375421    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.375881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.375862    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.394314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.394290    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdcl\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.395842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.395817    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.400645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.400614    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf672\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.401766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.401742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.470718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.470689    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.470884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.470798    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmf9\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.470884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.470851    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.470884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.470877    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471037    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471075    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471127    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471141    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471237    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471282    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.471820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.471788    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.472259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.472067    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.472259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.472084    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.472476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.472291    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.474322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.474296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.474665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.474645    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.487918 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.487892    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.488447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.488424    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmf9\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.492239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.492219    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:33.527940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.527909    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"]
Oct 31 00:17:33.528770 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:33.528743    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cedc8c4_4898_447c_8f36_75c1ab327b7a.slice/crio-0c7aed3a2968ef58bc782382322c9659202d750ee3e15c82889635d9d0c537ea WatchSource:0}: Error finding container 0c7aed3a2968ef58bc782382322c9659202d750ee3e15c82889635d9d0c537ea: Status 404 returned error can't find the container with id 0c7aed3a2968ef58bc782382322c9659202d750ee3e15c82889635d9d0c537ea
Oct 31 00:17:33.532202 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.532160    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:33.532312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.532249    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:33.532312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.532294    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:33.559840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.559739    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:33.607364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.607330    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:33.806448 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.806409    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"]
Oct 31 00:17:33.817051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.816497    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:33.817051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.816588    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:33.817051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.816628    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:33.891634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.891599    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"
Oct 31 00:17:33.892054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.891603    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8" event={"ID":"e2dab808-935c-4922-a6a7-407540897107","Type":"ContainerDied","Data":"e3a82d6632777c323decea568679d6ab2f035a87121226d4385a8ba85767cda4"}
Oct 31 00:17:33.892054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.891806    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a82d6632777c323decea568679d6ab2f035a87121226d4385a8ba85767cda4"
Oct 31 00:17:33.893898 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.893871    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" event={"ID":"9cedc8c4-4898-447c-8f36-75c1ab327b7a","Type":"ContainerStarted","Data":"d8ac09d1e03aa3d9c57a2c8d467120b068d0b5e3e223f94521eebe400c67967e"}
Oct 31 00:17:33.894005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.893923    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" event={"ID":"9cedc8c4-4898-447c-8f36-75c1ab327b7a","Type":"ContainerStarted","Data":"0c7aed3a2968ef58bc782382322c9659202d750ee3e15c82889635d9d0c537ea"}
Oct 31 00:17:33.940487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.939385    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podStartSLOduration=1.939363789 podStartE2EDuration="1.939363789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:33.938817972 +0000 UTC m=+1421.885188690" watchObservedRunningTime="2026-10-31 00:17:33.939363789 +0000 UTC m=+1421.885734508"
Oct 31 00:17:33.957596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.956779    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" event={"ID":"cf087db0-eaae-4c32-8e65-4a6238b2e282","Type":"ContainerStarted","Data":"fff985c9dadcc3cf20e28778344b3722fa112b688ce9b492f87878df1202a584"}
Oct 31 00:17:33.960497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.960020    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp" event={"ID":"e2ce096e-fcb0-471b-a842-382de2ba1100","Type":"ContainerDied","Data":"8030959d4ea452ab21373bb2ff2d1a0197b395f18d99a0c2b2a0954a66aa1069"}
Oct 31 00:17:33.960497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.960058    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8030959d4ea452ab21373bb2ff2d1a0197b395f18d99a0c2b2a0954a66aa1069"
Oct 31 00:17:33.960497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:33.960107    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"
Oct 31 00:17:34.009271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.009232    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:34.009735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.009705    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:34.009970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.009936    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:34.018609 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:34.018087    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda123a837_3763_4b5f_b18d_7281b1ebcc63.slice/crio-913ee7c128586603c32029942d58506f44fd1bf000461a09baf04ef283c2caa8 WatchSource:0}: Error finding container 913ee7c128586603c32029942d58506f44fd1bf000461a09baf04ef283c2caa8: Status 404 returned error can't find the container with id 913ee7c128586603c32029942d58506f44fd1bf000461a09baf04ef283c2caa8
Oct 31 00:17:34.036076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.033958    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"]
Oct 31 00:17:34.036076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.035712    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:34.036076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.035846    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:34.036076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.035935    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:17:34.049331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.048577    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"]
Oct 31 00:17:34.196947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.194737    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:34.196947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.194786    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:34.208551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:34.208529    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:35.356859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.355601    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:35.356859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.355666    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:35.442594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.438838    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" event={"ID":"a123a837-3763-4b5f-b18d-7281b1ebcc63","Type":"ContainerStarted","Data":"1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08"}
Oct 31 00:17:35.442594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.438911    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" event={"ID":"a123a837-3763-4b5f-b18d-7281b1ebcc63","Type":"ContainerStarted","Data":"913ee7c128586603c32029942d58506f44fd1bf000461a09baf04ef283c2caa8"}
Oct 31 00:17:35.527544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.521343    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" event={"ID":"54e1e46f-db63-441c-a3ce-161b24384674","Type":"ContainerStarted","Data":"a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e"}
Oct 31 00:17:35.527544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.521417    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" event={"ID":"54e1e46f-db63-441c-a3ce-161b24384674","Type":"ContainerStarted","Data":"1b4625576ba0aa8e81c8fe17ec72990f04ac6386270ecc7c6cdba5eeb2e92bb5"}
Oct 31 00:17:35.572681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.572647    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:35.573166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.573143    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:35.573327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.573304    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:35.622547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.613203    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:35.623628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.623596    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:35.624307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.624278    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:35.672504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.639240    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podStartSLOduration=2.63921832 podStartE2EDuration="2.63921832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:35.499194078 +0000 UTC m=+1423.445564796" watchObservedRunningTime="2026-10-31 00:17:35.63921832 +0000 UTC m=+1423.585589041"
Oct 31 00:17:35.672806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.649514    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" event={"ID":"cf087db0-eaae-4c32-8e65-4a6238b2e282","Type":"ContainerStarted","Data":"faba5592806f102eccc49d8f9f1b7675d239e628fa99db84b139e966000b332b"}
Oct 31 00:17:35.767909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.767847    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podStartSLOduration=2.767825528 podStartE2EDuration="2.767825528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:35.673127294 +0000 UTC m=+1423.619498011" watchObservedRunningTime="2026-10-31 00:17:35.767825528 +0000 UTC m=+1423.714196247"
Oct 31 00:17:35.874281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.874151    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podStartSLOduration=2.874129392 podStartE2EDuration="2.874129392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:35.763133762 +0000 UTC m=+1423.709504480" watchObservedRunningTime="2026-10-31 00:17:35.874129392 +0000 UTC m=+1423.820500111"
Oct 31 00:17:35.907411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.907365    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:35.965556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.965505    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:35.967004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.966972    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.000976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:35.999693    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.013160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.013124    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.136760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.135562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.136760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.136008    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:36.223175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.223126    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:36.223442 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.223407    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:36.253843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.247725    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.362487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.356538    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.504144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.504028    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:36.504419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.504374    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:36.504613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.504506    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:36.531130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.531083    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.565859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.565817    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:36.566089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.566064    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:36.603818 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:17:36.603781    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:17:36.609885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.609856    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:36.610125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.610085    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:36.762687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.762584    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.796203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.796149    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.797268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.796427    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.988602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.988556    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:36.998497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.989168    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:17:36.998497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.988762    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:36.998497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:36.991971    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:17:37.045591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.000748    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:37.045591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.001145    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:17:37.045591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.024669    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:37.072104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.055619    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:17:37.187484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.141633    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:37.187484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.142016    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:37.187484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.142661    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:17:37.187484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.142717    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:17:37.310549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.282605    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:37.310549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.282667    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:37.310549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.285442    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:37.310549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.285809    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:17:37.397489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.370275    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:37.417588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.390706    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:37.429487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.418191    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:17:37.494489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.493660    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:37.494489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.493716    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:37.549579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.539874    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:37.848902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.848852    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.009485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.968608    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:38.013584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.013542    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:38.021489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.017624    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:38.029484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.025528    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:38.129222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.013783    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:38.137556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.015121    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:17:38.141486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.137883    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:17:38.184489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.013690    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.192491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.013745    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:38.192491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.036391    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:38.224490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:37.983304    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.612487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.587683    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:38.640605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.640560    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.661170    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.661232    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.389560    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.663499    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.663557    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.603055    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.051378    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.383005    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:38.681957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.423834    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.729490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.217698    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:17:38.749487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.729838    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:17:38.843490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.049750    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:38.843490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.798212    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.079492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.078605    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 31 00:17:39.099800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:38.893386    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.100178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.100151    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:39.100389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.100357    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:39.157504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.137044    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.196981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.192112    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.197431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.192185    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.197798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.192210    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:39.236476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.236421    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:39.253002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.252956    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:39.558485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.554429    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.605106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.605069    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:39.612496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.605404    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:39.623673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.617894    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:39.623673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.617959    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:39.649250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.628814    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:39.649967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.649928    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:39.680825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.680741    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.681262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.681239    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:39.681626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.681530    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:39.681971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.681946    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:39.897546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:39.894599    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.053494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.047565    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:40.070867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.050715    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.137485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.124277    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.237779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.237667    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.242600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.242545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.243125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.243090    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.273605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.273556    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:40.539552 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:17:40.511094    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.732s"
Oct 31 00:17:40.777623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.777583    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:40.778204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.778169    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:40.810091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.809998    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:17:40.810392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.810368    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 31 00:17:40.811524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.811500    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:40.811687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.811665    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:40.816483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.814282    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.838993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.838960    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:40.839213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.839184    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:40.860391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.851366    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.869620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.869567    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.869770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.869699    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:40.869770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.869760    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:40.918507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.908885    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:40.922487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.919143    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:40.963563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:40.963511    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:41.089288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.089194    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:41.272161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.233628    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:41.272161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.234673    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:41.272161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.269273    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:41.272161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.267663    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:41.272161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.267613    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:41.272161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.267685    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:41.295918 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.295876    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:41.524439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.524404    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:41.524756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.524728    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:41.641529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.632592    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:41.641529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.632660    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:41.644228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.644191    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:41.644350 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.644250    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:41.841539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.822519    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:41.920499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.919881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:41.920884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.920851    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:41.924491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.921081    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:41.924491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.921268    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:41.964598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:41.964553    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:42.661278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.288660    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:42.669272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.290009    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:42.674315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.290042    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:42.679764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.290062    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:42.680111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.290081    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:42.680379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.303300    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:42.680621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.540712    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:42.680749 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.680723    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:42.680832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.657751    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:42.680931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.680917    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:42.759237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.759202    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:42.759543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.759517    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:42.798518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.792788    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:42.798518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.792850    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:42.810114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.810045    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:42.841378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.841308    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:42.920771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.920641    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:42.923679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.923635    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:42.945494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.937099    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:42.945494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:42.938254    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.205553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.197820    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:43.205553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.197885    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:43.599547 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:17:43.565931    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.312s"
Oct 31 00:17:43.625485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.577492    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:43.625485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.607244    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:43.625485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.585607    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:43.625485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.607337    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:43.643805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.627691    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:43.643805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.627759    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:43.666878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.655635    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:43.753721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.753683    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.764491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.754349    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:43.764491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.754732    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.764491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.755018    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.764491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.755324    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.764491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.755625    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.841932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.841891    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.866514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.866394    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:43.929497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.921534    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:43.929497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.926543    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:43.945980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.945937    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:43.946814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:43.946205    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.194226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.194189    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:44.194537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.194505    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:44.504005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.503911    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:44.505243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.505212    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:44.563488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.562892    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:44.563488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.562964    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:44.591490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.582652    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.613487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.607982    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:44.613487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.608038    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:44.658067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.658024    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.733999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.733858    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.757552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.756606    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.757552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.757018    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.757552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.757290    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.757552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.757547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.757906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.757771    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.761502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.758007    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.844660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.843724    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.924489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.922638    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:17:44.933510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.928547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.950497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.948095    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:44.968501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:44.967097    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.152897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.134782    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.286494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.262587    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:45.286494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.262651    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:45.553290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.549002    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:45.553290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.549063    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:45.617489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.610618    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:45.617489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.610684    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:45.687633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.687589    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.705689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.705655    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:45.705996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.705945    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:45.815550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.790392    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.816324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.793637    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:45.840491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.793697    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.841031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.793720    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.866789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.793762    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:45.899507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.793782    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:45.900081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.846053    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:45.900506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:45.899822    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 31 00:17:46.133554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.056057    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:46.215853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.212828    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.215853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.213096    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.238542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.228893    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:46.238542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.228947    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:46.283479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.283361    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.510499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.500531    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:17:46.510499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.500594    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:17:46.610490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.610025    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:46.610490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.610105    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:46.680355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.680319    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:46.680816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.680789    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:46.803626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.803504    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.816439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.816403    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:46.816813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.816782    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:46.842541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.842503    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.869921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.869846    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:46.871637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.871601    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.900609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.900566    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:46.901621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:46.900841    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.195045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.194988    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.197042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.196999    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:47.197246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.197220    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:47.217422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.217375    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.218032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.218000    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:47.302488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.283915    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:47.443492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.426620    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Oct 31 00:17:47.563698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.563598    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:47.564001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.563956    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:47.618154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.618110    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:17:47.618497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.618412    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:17:47.846825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.846720    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.851600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.847106    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:17:47.866500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.852139    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:47.904207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.903223    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.921523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.908516    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:47.937323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.919540    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.956070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.919580    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:47.963399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.919605    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:47.985530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:47.938426    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.202760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.202716    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.267488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.225982    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.399907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.228648    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:48.400660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.295060    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:17:48.465598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.452638    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:17:48.465598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.452697    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:17:48.574594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.574549    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:48.644966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.644913    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:48.645617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.620835    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:48.645863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.645848    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:48.646014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.646002    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:48.646157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.646146    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:48.658581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.644814    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:48.658804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.658779    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.682489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.677110    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.866732 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.866634    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.867980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.867562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.947531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.908703    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.949624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.949590    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.969699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.969648    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.971558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.971530    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:48.982603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.982568    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:17:48.987595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:48.987543    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.234489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.228786    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.401009 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.400960    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.428836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.421595    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:49.431517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.421658    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.548547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.547572    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.592515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.588143    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:49.592515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.588213    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:49.742138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.740829    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:49.742138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.740895    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.742138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.742115    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:49.742488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.742183    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.932490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.909185    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:49.977528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.970017    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:49.978888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.978837    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.008657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.980309    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.008657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.980661    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.008657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.980865    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.008657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.986534    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.008657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:49.988071    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.166491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.149239    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.265546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.230642    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.330526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.291795    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:50.330526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.291875    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.439606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.439558    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.441065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.441033    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.474571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.448978    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.688091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.688052    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 31 00:17:50.688452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.688418    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 31 00:17:50.717270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.717226    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.824852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.824815    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:50.825156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.825127    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.861667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.861631    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:50.861926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.861900    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.947544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.936951    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.977822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.949117    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:50.998493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.983714    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:50.998493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.984057    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.998493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.984307    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:17:50.998493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.984575    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:17:51.000651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:50.970554    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:17:51.003056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.002281    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:51.009489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.006449    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:51.013647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.011093    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:17:51.238683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.238022    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:17:51.334491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.326336    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:51.577546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.574341    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:17:51.628629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.628593    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:17:51.659803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.659720    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:17:51.790144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:51.786744    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:52.024899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.023047    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:52.024899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.023297    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:17:52.043218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.043177    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:52.043708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.043610    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:52.045100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.045070    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:17:52.057852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.057801    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:52.098502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.087306    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:17:52.119279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.119250    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:17:52.122767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.122731    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:52.146946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.146921    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:52.147498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.147452    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:52.147629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.147518    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:52.151365 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.151346    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:17:52.157144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.157125    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:17:52.158002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.157985    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:17:52.163485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.159565    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:17:52.163485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.159613    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:17:52.163853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.163835    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:52.170636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.169958    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:17:52.170941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.170925    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:17:52.171743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.171728    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:17:52.189077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.189055    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:17:52.260479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.255564    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:17:52.260756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:52.260738    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:17:54.228165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.228095    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"]
Oct 31 00:17:54.235396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.235373    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.236546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.236527    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"]
Oct 31 00:17:54.240165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.240142    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:54.240551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.240535    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:17:54.240979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.240928    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.245971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.245949    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"]
Oct 31 00:17:54.251808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.251787    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"]
Oct 31 00:17:54.259947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.259927    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"]
Oct 31 00:17:54.265829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.265812    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.266066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.266049    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"]
Oct 31 00:17:54.399167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399128    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bv5\" (UniqueName: \"kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.399358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399184    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzz9\" (UniqueName: \"kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.399358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399299    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.399358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399339    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.399588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399423    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.399647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399600    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.399806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399789    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.399862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399824    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.399980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.399965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtdm\" (UniqueName: \"kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.501962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.501877    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.502143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502030    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.502143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502054    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.502143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502140    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtdm\" (UniqueName: \"kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.502310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502197    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bv5\" (UniqueName: \"kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.502310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzz9\" (UniqueName: \"kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.502310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502257    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.502310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502287    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.502564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.502318    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.503299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.503267    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.503745 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.503716    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.505362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.505340    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.505563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.505364    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.505833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.505812    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.505911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.505842    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.544018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.543994    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzz9\" (UniqueName: \"kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9\") pod \"maas-api-revoke-sub-3041f7fd-k66fj\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.544189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.544167    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtdm\" (UniqueName: \"kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm\") pod \"maas-api-revoke-sub-c50d42ee-bwjc8\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.544407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.544387    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bv5\" (UniqueName: \"kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5\") pod \"maas-api-revoke-sub-6e1f9f39-crndx\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.555280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.555254    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:54.560965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.560943    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:54.592560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.592536    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:54.888973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.888944    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"]
Oct 31 00:17:54.890539 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:54.890511    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6cfda0_f8f3_44f2_b858_badbe40119c6.slice/crio-57681e98a9cb7f95fb1c6a8a8ebb39ee547dad6b2ca84776cfd3da347e5a0d68 WatchSource:0}: Error finding container 57681e98a9cb7f95fb1c6a8a8ebb39ee547dad6b2ca84776cfd3da347e5a0d68: Status 404 returned error can't find the container with id 57681e98a9cb7f95fb1c6a8a8ebb39ee547dad6b2ca84776cfd3da347e5a0d68
Oct 31 00:17:54.895502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.895475    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"]
Oct 31 00:17:54.941826 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:54.941795    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc912476_3494_48fa_be70_0156e78663b9.slice/crio-27fc93503cf5361c1ff279d1c8a1220ae2c5f6fc378730a473358f9ee2d80d07 WatchSource:0}: Error finding container 27fc93503cf5361c1ff279d1c8a1220ae2c5f6fc378730a473358f9ee2d80d07: Status 404 returned error can't find the container with id 27fc93503cf5361c1ff279d1c8a1220ae2c5f6fc378730a473358f9ee2d80d07
Oct 31 00:17:54.945391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:54.945368    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"]
Oct 31 00:17:55.311287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:55.311253    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj" event={"ID":"6e6cfda0-f8f3-44f2-b858-badbe40119c6","Type":"ContainerStarted","Data":"6d6b23c55690a60b85350b6c80f98d020a1f96cb17625c8d3abcaf3a6ad7abeb"}
Oct 31 00:17:55.311788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:55.311321    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj" event={"ID":"6e6cfda0-f8f3-44f2-b858-badbe40119c6","Type":"ContainerStarted","Data":"57681e98a9cb7f95fb1c6a8a8ebb39ee547dad6b2ca84776cfd3da347e5a0d68"}
Oct 31 00:17:55.312836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:55.312814    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8" event={"ID":"bc912476-3494-48fa-be70-0156e78663b9","Type":"ContainerStarted","Data":"27fc93503cf5361c1ff279d1c8a1220ae2c5f6fc378730a473358f9ee2d80d07"}
Oct 31 00:17:55.314390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:55.314368    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx" event={"ID":"026df7e1-ad6c-4ee6-bf8e-26585908b64c","Type":"ContainerStarted","Data":"fb6539f8a476985b59cb639a9b97e48172f0ad1a71ef2342378bc86dc46b3ed3"}
Oct 31 00:17:55.336824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:55.336785    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj" podStartSLOduration=2.336772018 podStartE2EDuration="2.336772018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:55.332912419 +0000 UTC m=+1443.279283137" watchObservedRunningTime="2026-10-31 00:17:55.336772018 +0000 UTC m=+1443.283142773"
Oct 31 00:17:56.342249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:56.342200    5078 generic.go:356] "Generic (PLEG): container finished" podID="026df7e1-ad6c-4ee6-bf8e-26585908b64c" containerID="d6f22c8947f4ba632b4c905562014ffc057985cf4c0554e3b8ce78979819ee78" exitCode=0
Oct 31 00:17:56.342787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:56.342362    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx" event={"ID":"026df7e1-ad6c-4ee6-bf8e-26585908b64c","Type":"ContainerDied","Data":"d6f22c8947f4ba632b4c905562014ffc057985cf4c0554e3b8ce78979819ee78"}
Oct 31 00:17:56.346808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:56.346765    5078 generic.go:356] "Generic (PLEG): container finished" podID="6e6cfda0-f8f3-44f2-b858-badbe40119c6" containerID="6d6b23c55690a60b85350b6c80f98d020a1f96cb17625c8d3abcaf3a6ad7abeb" exitCode=0
Oct 31 00:17:56.347055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:56.347028    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj" event={"ID":"6e6cfda0-f8f3-44f2-b858-badbe40119c6","Type":"ContainerDied","Data":"6d6b23c55690a60b85350b6c80f98d020a1f96cb17625c8d3abcaf3a6ad7abeb"}
Oct 31 00:17:56.349997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:56.349929    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8" event={"ID":"bc912476-3494-48fa-be70-0156e78663b9","Type":"ContainerStarted","Data":"b57a345318ef1a992828c427448522bf21a9f3cd277f300068019935f45cfb1a"}
Oct 31 00:17:57.361895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.361862    5078 generic.go:356] "Generic (PLEG): container finished" podID="bc912476-3494-48fa-be70-0156e78663b9" containerID="b57a345318ef1a992828c427448522bf21a9f3cd277f300068019935f45cfb1a" exitCode=0
Oct 31 00:17:57.362336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.361957    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8" event={"ID":"bc912476-3494-48fa-be70-0156e78663b9","Type":"ContainerDied","Data":"b57a345318ef1a992828c427448522bf21a9f3cd277f300068019935f45cfb1a"}
Oct 31 00:17:57.625531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.622177    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:57.719962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.719924    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca\") pod \"bc912476-3494-48fa-be70-0156e78663b9\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") "
Oct 31 00:17:57.720138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.720014    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca\") pod \"bc912476-3494-48fa-be70-0156e78663b9\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") "
Oct 31 00:17:57.720201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.720177    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm\" (UniqueName: \"kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm\") pod \"bc912476-3494-48fa-be70-0156e78663b9\" (UID: \"bc912476-3494-48fa-be70-0156e78663b9\") "
Oct 31 00:17:57.721583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.721544    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca" pod "bc912476-3494-48fa-be70-0156e78663b9" (UID: "bc912476-3494-48fa-be70-0156e78663b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:57.725130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.724876    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm" pod "bc912476-3494-48fa-be70-0156e78663b9" (UID: "bc912476-3494-48fa-be70-0156e78663b9"). InnerVolumeSpecName "kube-api-access-lqtdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:57.726064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.726038    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca" pod "bc912476-3494-48fa-be70-0156e78663b9" (UID: "bc912476-3494-48fa-be70-0156e78663b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:57.730834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.730646    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:57.760531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.760502    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:57.822151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.822115    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca\") pod \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") "
Oct 31 00:17:57.822330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.822194    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5\" (UniqueName: \"kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5\") pod \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") "
Oct 31 00:17:57.822421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.822396    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca\") pod \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\" (UID: \"026df7e1-ad6c-4ee6-bf8e-26585908b64c\") "
Oct 31 00:17:57.823168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.823141    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca" pod "026df7e1-ad6c-4ee6-bf8e-26585908b64c" (UID: "026df7e1-ad6c-4ee6-bf8e-26585908b64c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:57.823279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.823247    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtdm\" (UniqueName: \"kubernetes.io/projected/bc912476-3494-48fa-be70-0156e78663b9-kube-api-access-lqtdm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.823279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.823260    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.823279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.823270    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc912476-3494-48fa-be70-0156e78663b9-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.823279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.823279    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc912476-3494-48fa-be70-0156e78663b9-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.825702 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.825672    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5" pod "026df7e1-ad6c-4ee6-bf8e-26585908b64c" (UID: "026df7e1-ad6c-4ee6-bf8e-26585908b64c"). InnerVolumeSpecName "kube-api-access-h4bv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:57.826286 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.826266    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca" pod "026df7e1-ad6c-4ee6-bf8e-26585908b64c" (UID: "026df7e1-ad6c-4ee6-bf8e-26585908b64c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:57.924887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.924856    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca\") pod \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") "
Oct 31 00:17:57.925030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.925005    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9\" (UniqueName: \"kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9\") pod \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") "
Oct 31 00:17:57.925077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.925067    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca\") pod \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\" (UID: \"6e6cfda0-f8f3-44f2-b858-badbe40119c6\") "
Oct 31 00:17:57.925278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.925258    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca" pod "6e6cfda0-f8f3-44f2-b858-badbe40119c6" (UID: "6e6cfda0-f8f3-44f2-b858-badbe40119c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:57.925740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.925666    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.925740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.925692    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026df7e1-ad6c-4ee6-bf8e-26585908b64c-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.925740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.925706    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4bv5\" (UniqueName: \"kubernetes.io/projected/026df7e1-ad6c-4ee6-bf8e-26585908b64c-kube-api-access-h4bv5\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:57.927334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.927307    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca" pod "6e6cfda0-f8f3-44f2-b858-badbe40119c6" (UID: "6e6cfda0-f8f3-44f2-b858-badbe40119c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:57.927334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:57.927315    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9" pod "6e6cfda0-f8f3-44f2-b858-badbe40119c6" (UID: "6e6cfda0-f8f3-44f2-b858-badbe40119c6"). InnerVolumeSpecName "kube-api-access-kzzz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:58.027753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.027720    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzz9\" (UniqueName: \"kubernetes.io/projected/6e6cfda0-f8f3-44f2-b858-badbe40119c6-kube-api-access-kzzz9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:58.027753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.027752    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6cfda0-f8f3-44f2-b858-badbe40119c6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:58.220704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.220671    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"]
Oct 31 00:17:58.221866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.221843    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bc912476-3494-48fa-be70-0156e78663b9" containerName="revoke-keys"
Oct 31 00:17:58.221866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.221866    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc912476-3494-48fa-be70-0156e78663b9" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.221896    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6e6cfda0-f8f3-44f2-b858-badbe40119c6" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.221903    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6cfda0-f8f3-44f2-b858-badbe40119c6" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.221923    5078 cpu_manager.go:414] "containerMap: removing container" podUID="026df7e1-ad6c-4ee6-bf8e-26585908b64c" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.221931    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026df7e1-ad6c-4ee6-bf8e-26585908b64c" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.222080    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="026df7e1-ad6c-4ee6-bf8e-26585908b64c" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.222111    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc912476-3494-48fa-be70-0156e78663b9" containerName="revoke-keys"
Oct 31 00:17:58.223031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.222183    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6cfda0-f8f3-44f2-b858-badbe40119c6" containerName="revoke-keys"
Oct 31 00:17:58.227263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.227239    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.229345 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.229318    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"]
Oct 31 00:17:58.331779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.331736    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.331945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.331836    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.332178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.332160    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbp7\" (UniqueName: \"kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.374064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.374028    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx" event={"ID":"026df7e1-ad6c-4ee6-bf8e-26585908b64c","Type":"ContainerDied","Data":"fb6539f8a476985b59cb639a9b97e48172f0ad1a71ef2342378bc86dc46b3ed3"}
Oct 31 00:17:58.374577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.374096    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb6539f8a476985b59cb639a9b97e48172f0ad1a71ef2342378bc86dc46b3ed3"
Oct 31 00:17:58.374577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.374179    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"
Oct 31 00:17:58.376349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.376324    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"
Oct 31 00:17:58.376515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.376345    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj" event={"ID":"6e6cfda0-f8f3-44f2-b858-badbe40119c6","Type":"ContainerDied","Data":"57681e98a9cb7f95fb1c6a8a8ebb39ee547dad6b2ca84776cfd3da347e5a0d68"}
Oct 31 00:17:58.376515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.376396    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57681e98a9cb7f95fb1c6a8a8ebb39ee547dad6b2ca84776cfd3da347e5a0d68"
Oct 31 00:17:58.378220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.378201    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"
Oct 31 00:17:58.378307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.378271    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8" event={"ID":"bc912476-3494-48fa-be70-0156e78663b9","Type":"ContainerDied","Data":"27fc93503cf5361c1ff279d1c8a1220ae2c5f6fc378730a473358f9ee2d80d07"}
Oct 31 00:17:58.378363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.378308    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27fc93503cf5361c1ff279d1c8a1220ae2c5f6fc378730a473358f9ee2d80d07"
Oct 31 00:17:58.435279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.435246    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbp7\" (UniqueName: \"kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.436912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.436883    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.440175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.440151    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.441198 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.440076    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.449612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.444966    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.453503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.453425    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbp7\" (UniqueName: \"kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7\") pod \"maas-api-revoke-sub-76b4292c-zg9rh\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.547296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.547216    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:17:58.840310 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:17:58.840265    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd084ee93_37d1_4c1b_9dfa_b86b61063ae8.slice/crio-c51a6ac1a3a13a36d21be04c6e650fbfdff7de5b55fddb4e4ac016c508ca9047 WatchSource:0}: Error finding container c51a6ac1a3a13a36d21be04c6e650fbfdff7de5b55fddb4e4ac016c508ca9047: Status 404 returned error can't find the container with id c51a6ac1a3a13a36d21be04c6e650fbfdff7de5b55fddb4e4ac016c508ca9047
Oct 31 00:17:58.842959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:58.842937    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"]
Oct 31 00:17:59.389886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:59.389849    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh" event={"ID":"d084ee93-37d1-4c1b-9dfa-b86b61063ae8","Type":"ContainerStarted","Data":"a2373afc0eedb17a210ae8f2da2df629150dc2a353f14856f4e41641774cd329"}
Oct 31 00:17:59.390327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:59.389917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh" event={"ID":"d084ee93-37d1-4c1b-9dfa-b86b61063ae8","Type":"ContainerStarted","Data":"c51a6ac1a3a13a36d21be04c6e650fbfdff7de5b55fddb4e4ac016c508ca9047"}
Oct 31 00:17:59.422600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:17:59.422540    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh" podStartSLOduration=1.422525676 podStartE2EDuration="1.422525676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:59.417477905 +0000 UTC m=+1447.363848635" watchObservedRunningTime="2026-10-31 00:17:59.422525676 +0000 UTC m=+1447.368896405"
Oct 31 00:18:00.404307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:00.404269    5078 generic.go:356] "Generic (PLEG): container finished" podID="d084ee93-37d1-4c1b-9dfa-b86b61063ae8" containerID="a2373afc0eedb17a210ae8f2da2df629150dc2a353f14856f4e41641774cd329" exitCode=0
Oct 31 00:18:00.404821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:00.404409    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh" event={"ID":"d084ee93-37d1-4c1b-9dfa-b86b61063ae8","Type":"ContainerDied","Data":"a2373afc0eedb17a210ae8f2da2df629150dc2a353f14856f4e41641774cd329"}
Oct 31 00:18:01.609053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.609025    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:18:01.698207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.698104    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca\") pod \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") "
Oct 31 00:18:01.698392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.698221    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7\" (UniqueName: \"kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7\") pod \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") "
Oct 31 00:18:01.698784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.698750    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca\") pod \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\" (UID: \"d084ee93-37d1-4c1b-9dfa-b86b61063ae8\") "
Oct 31 00:18:01.698897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.698810    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca" pod "d084ee93-37d1-4c1b-9dfa-b86b61063ae8" (UID: "d084ee93-37d1-4c1b-9dfa-b86b61063ae8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:01.700840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.700805    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:01.703006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.702936    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7" pod "d084ee93-37d1-4c1b-9dfa-b86b61063ae8" (UID: "d084ee93-37d1-4c1b-9dfa-b86b61063ae8"). InnerVolumeSpecName "kube-api-access-fxbp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:01.703402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.703376    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca" pod "d084ee93-37d1-4c1b-9dfa-b86b61063ae8" (UID: "d084ee93-37d1-4c1b-9dfa-b86b61063ae8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:01.803200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.803162    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbp7\" (UniqueName: \"kubernetes.io/projected/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-kube-api-access-fxbp7\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:01.803200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:01.803198    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d084ee93-37d1-4c1b-9dfa-b86b61063ae8-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:02.422975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.422951    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"
Oct 31 00:18:02.423148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.422982    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh" event={"ID":"d084ee93-37d1-4c1b-9dfa-b86b61063ae8","Type":"ContainerDied","Data":"c51a6ac1a3a13a36d21be04c6e650fbfdff7de5b55fddb4e4ac016c508ca9047"}
Oct 31 00:18:02.423148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.423028    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51a6ac1a3a13a36d21be04c6e650fbfdff7de5b55fddb4e4ac016c508ca9047"
Oct 31 00:18:02.795844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.795125    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"]
Oct 31 00:18:02.796723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.796624    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d084ee93-37d1-4c1b-9dfa-b86b61063ae8" containerName="revoke-keys"
Oct 31 00:18:02.796723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.796649    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d084ee93-37d1-4c1b-9dfa-b86b61063ae8" containerName="revoke-keys"
Oct 31 00:18:02.796925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.796897    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d084ee93-37d1-4c1b-9dfa-b86b61063ae8" containerName="revoke-keys"
Oct 31 00:18:02.801495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.801455    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.819236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.819201    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"]
Oct 31 00:18:02.937781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.937743    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69phb\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.937967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.937804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.937967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.937834    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.937967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.937914    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.938080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.938012    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.938080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.938058    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.938149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.938081    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.938179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.938150    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:02.938251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:02.938233    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040209    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040245    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040446 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040292    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040446 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040366    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040527    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69phb\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040571    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040596    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040632    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040680    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040723    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.040765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.040743    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.041086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.041066    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.041153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.041109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.041441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.041421    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.043012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.042988    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.043137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.043119    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.065454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.065383    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.065883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.065858    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69phb\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.227135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.227103    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:03.527300 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:03.527260    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865f19cd_c107_4a5c_8ec6_64566ede3bf4.slice/crio-f9abb3a4e6d531128c089c672f82a6a74ea8107d5f6c919c2d7b9cfb86e776ec WatchSource:0}: Error finding container f9abb3a4e6d531128c089c672f82a6a74ea8107d5f6c919c2d7b9cfb86e776ec: Status 404 returned error can't find the container with id f9abb3a4e6d531128c089c672f82a6a74ea8107d5f6c919c2d7b9cfb86e776ec
Oct 31 00:18:03.529243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.529133    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"]
Oct 31 00:18:03.530423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.530389    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:18:03.530531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.530505    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:18:03.530603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:03.530546    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:18:04.293321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.291236    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"]
Oct 31 00:18:04.351486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.351383    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"]
Oct 31 00:18:04.356868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.351507    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.368547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.368175    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:04.381489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.370104    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:18:04.444662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.444623    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" event={"ID":"865f19cd-c107-4a5c-8ec6-64566ede3bf4","Type":"ContainerStarted","Data":"bc55c8c35ae3edb7c71643b0df613a0601f44d9e81e174ec5553e58e5d6354db"}
Oct 31 00:18:04.444833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.444694    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" event={"ID":"865f19cd-c107-4a5c-8ec6-64566ede3bf4","Type":"ContainerStarted","Data":"f9abb3a4e6d531128c089c672f82a6a74ea8107d5f6c919c2d7b9cfb86e776ec"}
Oct 31 00:18:04.468694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.468666    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.516205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.515868    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.516408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.516246    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5s8s\" (UniqueName: \"kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.517653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.517057    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podStartSLOduration=2.517039302 podStartE2EDuration="2.517039302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:04.495136713 +0000 UTC m=+1452.441507432" watchObservedRunningTime="2026-10-31 00:18:04.517039302 +0000 UTC m=+1452.463410023"
Oct 31 00:18:04.618379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.618283    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5s8s\" (UniqueName: \"kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.618582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.618492    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.618582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.618566    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.619421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.619388    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.623698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.623664    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.632071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.632050    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5s8s\" (UniqueName: \"kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s\") pod \"maas-api-revoke-sub-e784d715-njbwf\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:04.731768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:04.731739    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:05.149780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:05.149751    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"]
Oct 31 00:18:05.240606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:05.238156    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:05.251015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:05.244196    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:18:05.251015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:05.244248    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:18:05.519511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:05.494969    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf" event={"ID":"6bf6bac6-e10a-46a4-9a49-d97dfff92d51","Type":"ContainerStarted","Data":"29ae1e8747be9e420c9a4fcdd7b3b360075aff7b5865d0a001575aec784f08ec"}
Oct 31 00:18:06.227904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:06.227833    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:18:06.228076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:06.227918    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:18:06.448813 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:18:06.448770    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:18:06.517538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:06.516997    5078 generic.go:356] "Generic (PLEG): container finished" podID="6bf6bac6-e10a-46a4-9a49-d97dfff92d51" containerID="2b1c8341be38bd226248a87a71449b68631fb0f10eb0a625b3e7d1f7121ac158" exitCode=0
Oct 31 00:18:06.518290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:06.518268    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf" event={"ID":"6bf6bac6-e10a-46a4-9a49-d97dfff92d51","Type":"ContainerDied","Data":"2b1c8341be38bd226248a87a71449b68631fb0f10eb0a625b3e7d1f7121ac158"}
Oct 31 00:18:07.227828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.227791    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:18:07.228275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.227851    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:18:07.748310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.747390    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"]
Oct 31 00:18:07.752498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.752426    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.756389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.756349    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"]
Oct 31 00:18:07.849055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.848836    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p598\" (UniqueName: \"kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.849055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.848985    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.849587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.849561    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.855352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.855329    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:07.951358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.951328    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca\") pod \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") "
Oct 31 00:18:07.951535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.951430    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s\" (UniqueName: \"kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s\") pod \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") "
Oct 31 00:18:07.951593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.951582    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca\") pod \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\" (UID: \"6bf6bac6-e10a-46a4-9a49-d97dfff92d51\") "
Oct 31 00:18:07.951937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.951917    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p598\" (UniqueName: \"kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.952028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.952009    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.952129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.952114    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.953193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.952826    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.953193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.952894    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca" pod "6bf6bac6-e10a-46a4-9a49-d97dfff92d51" (UID: "6bf6bac6-e10a-46a4-9a49-d97dfff92d51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:07.955956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.955931    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s" pod "6bf6bac6-e10a-46a4-9a49-d97dfff92d51" (UID: "6bf6bac6-e10a-46a4-9a49-d97dfff92d51"). InnerVolumeSpecName "kube-api-access-v5s8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:07.956611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.956227    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca" pod "6bf6bac6-e10a-46a4-9a49-d97dfff92d51" (UID: "6bf6bac6-e10a-46a4-9a49-d97dfff92d51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:07.956611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.956568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:07.968106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:07.967901    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p598\" (UniqueName: \"kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598\") pod \"maas-api-revoke-sub-e480172b-sd5md\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:08.054250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.054216    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:08.054250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.054251    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5s8s\" (UniqueName: \"kubernetes.io/projected/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-kube-api-access-v5s8s\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:08.054456 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.054268    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6bac6-e10a-46a4-9a49-d97dfff92d51-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:08.124602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.124521    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:08.235192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.235163    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:18:08.235594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.235218    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:18:08.513259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.512209    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"]
Oct 31 00:18:08.560365 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.560332    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md" event={"ID":"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2","Type":"ContainerStarted","Data":"53400c5b1fc7279f7ea51228d8ddb2c81c20f3e0152038305dae9fcb0eb630ae"}
Oct 31 00:18:08.562723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.561989    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf" event={"ID":"6bf6bac6-e10a-46a4-9a49-d97dfff92d51","Type":"ContainerDied","Data":"29ae1e8747be9e420c9a4fcdd7b3b360075aff7b5865d0a001575aec784f08ec"}
Oct 31 00:18:08.562723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.562010    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"
Oct 31 00:18:08.562723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:08.562033    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ae1e8747be9e420c9a4fcdd7b3b360075aff7b5865d0a001575aec784f08ec"
Oct 31 00:18:09.227665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.227629    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:18:09.227842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.227681    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:18:09.596598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.595527    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md" event={"ID":"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2","Type":"ContainerStarted","Data":"374131d55b8cb84713c86de99521fd24a4833fa16ab1a514d8419fde6918e103"}
Oct 31 00:18:09.611284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.611221    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md" podStartSLOduration=2.611187444 podStartE2EDuration="2.611187444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:09.60682569 +0000 UTC m=+1457.553196407" watchObservedRunningTime="2026-10-31 00:18:09.611187444 +0000 UTC m=+1457.557558162"
Oct 31 00:18:09.824375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.824339    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"]
Oct 31 00:18:09.825154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.825134    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6bf6bac6-e10a-46a4-9a49-d97dfff92d51" containerName="revoke-keys"
Oct 31 00:18:09.825154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.825152    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf6bac6-e10a-46a4-9a49-d97dfff92d51" containerName="revoke-keys"
Oct 31 00:18:09.825334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.825302    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf6bac6-e10a-46a4-9a49-d97dfff92d51" containerName="revoke-keys"
Oct 31 00:18:09.829281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.829261    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:09.838099 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.838054    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"]
Oct 31 00:18:09.923969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.923709    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99qx\" (UniqueName: \"kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:09.923969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.923750    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:09.923969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:09.923937    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.025291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.025257    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99qx\" (UniqueName: \"kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.025540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.025300    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.025627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.025599    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.026269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.026249    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.028365 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.028339    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.043876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.043853    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99qx\" (UniqueName: \"kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx\") pod \"maas-api-revoke-sub-af0428ea-sqkdl\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.155323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.155287    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:10.228756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.228299    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:18:10.228756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.228357    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:18:10.477791 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:10.477755    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f171633_a74d_4ec9_a7c3_412672ba9f49.slice/crio-616985529bcb5dcc462554a2ef2cbf8bccd9dd19e756250fccbc4bd8b3355414 WatchSource:0}: Error finding container 616985529bcb5dcc462554a2ef2cbf8bccd9dd19e756250fccbc4bd8b3355414: Status 404 returned error can't find the container with id 616985529bcb5dcc462554a2ef2cbf8bccd9dd19e756250fccbc4bd8b3355414
Oct 31 00:18:10.480528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.480498    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"]
Oct 31 00:18:10.587918 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.587881    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl" event={"ID":"1f171633-a74d-4ec9-a7c3-412672ba9f49","Type":"ContainerStarted","Data":"616985529bcb5dcc462554a2ef2cbf8bccd9dd19e756250fccbc4bd8b3355414"}
Oct 31 00:18:10.590259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.590223    5078 generic.go:356] "Generic (PLEG): container finished" podID="1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" containerID="374131d55b8cb84713c86de99521fd24a4833fa16ab1a514d8419fde6918e103" exitCode=0
Oct 31 00:18:10.590444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:10.590411    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md" event={"ID":"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2","Type":"ContainerDied","Data":"374131d55b8cb84713c86de99521fd24a4833fa16ab1a514d8419fde6918e103"}
Oct 31 00:18:11.266993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.266911    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:11.269280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.269242    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:11.277784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.275809    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:18:11.633377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.630145    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl" event={"ID":"1f171633-a74d-4ec9-a7c3-412672ba9f49","Type":"ContainerStarted","Data":"ed36dbb828c436d91311aac26b78bfe9f0d29c3af752b57aa64603622c20d8ad"}
Oct 31 00:18:11.633377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.630250    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"]
Oct 31 00:18:11.644389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.644301    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"]
Oct 31 00:18:11.644640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.644423    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.672588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.672530    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl" podStartSLOduration=2.672511627 podStartE2EDuration="2.672511627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:11.665806832 +0000 UTC m=+1459.612177550" watchObservedRunningTime="2026-10-31 00:18:11.672511627 +0000 UTC m=+1459.618882345"
Oct 31 00:18:11.805919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.805675    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwpx\" (UniqueName: \"kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.805919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.805870    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.806146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.806027    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.907817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.907776    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.908750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.908703    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwpx\" (UniqueName: \"kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.908996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.908982    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.909955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.909934    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.912506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.912430    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.922632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.922609    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwpx\" (UniqueName: \"kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx\") pod \"maas-api-revoke-sub-2fe0ee14-mftlj\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:11.958752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.958721    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:11.976790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:11.976759    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:12.033971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.033931    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"]
Oct 31 00:18:12.035199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.035150    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" containerName="revoke-keys"
Oct 31 00:18:12.035199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.035180    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" containerName="revoke-keys"
Oct 31 00:18:12.035454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.035435    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" containerName="revoke-keys"
Oct 31 00:18:12.042308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.042280    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.064068 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.064042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"]
Oct 31 00:18:12.116485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.113305    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca\") pod \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") "
Oct 31 00:18:12.116485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.113621    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca\") pod \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") "
Oct 31 00:18:12.116485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.113738    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598\" (UniqueName: \"kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598\") pod \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\" (UID: \"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2\") "
Oct 31 00:18:12.116485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.114069    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca" pod "1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" (UID: "1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:12.116485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.114550    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:12.122256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.118927    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598" pod "1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" (UID: "1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2"). InnerVolumeSpecName "kube-api-access-2p598". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:12.122256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.119317    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca" pod "1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" (UID: "1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:12.216250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.215885    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.216250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.215950    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6scp\" (UniqueName: \"kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.218307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.216549    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.218307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.216672    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p598\" (UniqueName: \"kubernetes.io/projected/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-kube-api-access-2p598\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:12.218307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.216690    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:12.308247 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:12.308205    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf640ab64_980a_490e_a79b_43bcee5fc7ca.slice/crio-5e0ef18eef4d91f383a75e0c15437c8731745d31ecbae66c01d4940f76db2202 WatchSource:0}: Error finding container 5e0ef18eef4d91f383a75e0c15437c8731745d31ecbae66c01d4940f76db2202: Status 404 returned error can't find the container with id 5e0ef18eef4d91f383a75e0c15437c8731745d31ecbae66c01d4940f76db2202
Oct 31 00:18:12.310276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.310252    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"]
Oct 31 00:18:12.318662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.318634    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.319289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.319268    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6scp\" (UniqueName: \"kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.319827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.319804    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.320587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.320565    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.321876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.321856    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.332662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.332636    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6scp\" (UniqueName: \"kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp\") pod \"maas-api-revoke-sub-0229a4a6-5snnb\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.390106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.390045    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:12.638374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.638342    5078 generic.go:356] "Generic (PLEG): container finished" podID="1f171633-a74d-4ec9-a7c3-412672ba9f49" containerID="ed36dbb828c436d91311aac26b78bfe9f0d29c3af752b57aa64603622c20d8ad" exitCode=0
Oct 31 00:18:12.638581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.638487    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl" event={"ID":"1f171633-a74d-4ec9-a7c3-412672ba9f49","Type":"ContainerDied","Data":"ed36dbb828c436d91311aac26b78bfe9f0d29c3af752b57aa64603622c20d8ad"}
Oct 31 00:18:12.640095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.640078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"
Oct 31 00:18:12.640226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.640108    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md" event={"ID":"1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2","Type":"ContainerDied","Data":"53400c5b1fc7279f7ea51228d8ddb2c81c20f3e0152038305dae9fcb0eb630ae"}
Oct 31 00:18:12.640226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.640155    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53400c5b1fc7279f7ea51228d8ddb2c81c20f3e0152038305dae9fcb0eb630ae"
Oct 31 00:18:12.641483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.641438    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj" event={"ID":"f640ab64-980a-490e-a79b-43bcee5fc7ca","Type":"ContainerStarted","Data":"5e0ef18eef4d91f383a75e0c15437c8731745d31ecbae66c01d4940f76db2202"}
Oct 31 00:18:12.731998 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:12.731966    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba29423d_2585_4527_bd0f_ceee7ec96593.slice/crio-a0151cf2ab1d2333297f6cd4eefc8202f2a43eac328d667e10993988b5285ea9 WatchSource:0}: Error finding container a0151cf2ab1d2333297f6cd4eefc8202f2a43eac328d667e10993988b5285ea9: Status 404 returned error can't find the container with id a0151cf2ab1d2333297f6cd4eefc8202f2a43eac328d667e10993988b5285ea9
Oct 31 00:18:12.733768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:12.733704    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"]
Oct 31 00:18:13.652303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.652268    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb" event={"ID":"ba29423d-2585-4527-bd0f-ceee7ec96593","Type":"ContainerStarted","Data":"275609d3f15a6755e97b12baeff564cc97bf4c7680e254983699f1095356860a"}
Oct 31 00:18:13.652716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.652356    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb" event={"ID":"ba29423d-2585-4527-bd0f-ceee7ec96593","Type":"ContainerStarted","Data":"a0151cf2ab1d2333297f6cd4eefc8202f2a43eac328d667e10993988b5285ea9"}
Oct 31 00:18:13.654187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.654158    5078 generic.go:356] "Generic (PLEG): container finished" podID="f640ab64-980a-490e-a79b-43bcee5fc7ca" containerID="a06149d77879cae310393704e73664d155ed8c36322387ba5f6c525a55cebbd5" exitCode=0
Oct 31 00:18:13.654292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.654232    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj" event={"ID":"f640ab64-980a-490e-a79b-43bcee5fc7ca","Type":"ContainerDied","Data":"a06149d77879cae310393704e73664d155ed8c36322387ba5f6c525a55cebbd5"}
Oct 31 00:18:13.677017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.676970    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb" podStartSLOduration=1.676956018 podStartE2EDuration="1.676956018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:13.674150235 +0000 UTC m=+1461.620520953" watchObservedRunningTime="2026-10-31 00:18:13.676956018 +0000 UTC m=+1461.623326734"
Oct 31 00:18:13.867852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.867825    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:13.960375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.960335    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca\") pod \"1f171633-a74d-4ec9-a7c3-412672ba9f49\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") "
Oct 31 00:18:13.960608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.960543    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx\" (UniqueName: \"kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx\") pod \"1f171633-a74d-4ec9-a7c3-412672ba9f49\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") "
Oct 31 00:18:13.960695 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.960653    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca\") pod \"1f171633-a74d-4ec9-a7c3-412672ba9f49\" (UID: \"1f171633-a74d-4ec9-a7c3-412672ba9f49\") "
Oct 31 00:18:13.962123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.962084    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca" pod "1f171633-a74d-4ec9-a7c3-412672ba9f49" (UID: "1f171633-a74d-4ec9-a7c3-412672ba9f49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:13.963970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.963943    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx" pod "1f171633-a74d-4ec9-a7c3-412672ba9f49" (UID: "1f171633-a74d-4ec9-a7c3-412672ba9f49"). InnerVolumeSpecName "kube-api-access-c99qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:13.964100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:13.964075    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca" pod "1f171633-a74d-4ec9-a7c3-412672ba9f49" (UID: "1f171633-a74d-4ec9-a7c3-412672ba9f49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:14.079291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.079250    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.079291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.079292    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99qx\" (UniqueName: \"kubernetes.io/projected/1f171633-a74d-4ec9-a7c3-412672ba9f49-kube-api-access-c99qx\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.079588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.079310    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f171633-a74d-4ec9-a7c3-412672ba9f49-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.708499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.707966    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl" event={"ID":"1f171633-a74d-4ec9-a7c3-412672ba9f49","Type":"ContainerDied","Data":"616985529bcb5dcc462554a2ef2cbf8bccd9dd19e756250fccbc4bd8b3355414"}
Oct 31 00:18:14.708499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.708030    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="616985529bcb5dcc462554a2ef2cbf8bccd9dd19e756250fccbc4bd8b3355414"
Oct 31 00:18:14.708499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.708101    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"
Oct 31 00:18:14.719585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.712750    5078 generic.go:356] "Generic (PLEG): container finished" podID="ba29423d-2585-4527-bd0f-ceee7ec96593" containerID="275609d3f15a6755e97b12baeff564cc97bf4c7680e254983699f1095356860a" exitCode=0
Oct 31 00:18:14.719585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:14.713191    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb" event={"ID":"ba29423d-2585-4527-bd0f-ceee7ec96593","Type":"ContainerDied","Data":"275609d3f15a6755e97b12baeff564cc97bf4c7680e254983699f1095356860a"}
Oct 31 00:18:16.093940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.050772    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:16.206491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.182204    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca\") pod \"f640ab64-980a-490e-a79b-43bcee5fc7ca\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") "
Oct 31 00:18:16.206491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.182407    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca\") pod \"f640ab64-980a-490e-a79b-43bcee5fc7ca\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") "
Oct 31 00:18:16.206491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.182544    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx\" (UniqueName: \"kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx\") pod \"f640ab64-980a-490e-a79b-43bcee5fc7ca\" (UID: \"f640ab64-980a-490e-a79b-43bcee5fc7ca\") "
Oct 31 00:18:16.236489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.235563    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:16.668519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.636078    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca" pod "f640ab64-980a-490e-a79b-43bcee5fc7ca" (UID: "f640ab64-980a-490e-a79b-43bcee5fc7ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.723287    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx" pod "f640ab64-980a-490e-a79b-43bcee5fc7ca" (UID: "f640ab64-980a-490e-a79b-43bcee5fc7ca"). InnerVolumeSpecName "kube-api-access-znwpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.697705    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.697637    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.703741    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.705371    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.705630    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.722598    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:16.910237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.894026    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:16.997485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:16.987569    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:18:17.114491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.108035    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"
Oct 31 00:18:17.162534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.121267    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:17.162534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.122440    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:18:17.167921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.167071    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znwpx\" (UniqueName: \"kubernetes.io/projected/f640ab64-980a-490e-a79b-43bcee5fc7ca-kube-api-access-znwpx\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:17.167921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.167412    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:17.190079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.190019    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:17.304523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.266310    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:17.452184 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.452088    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca" pod "f640ab64-980a-490e-a79b-43bcee5fc7ca" (UID: "f640ab64-980a-490e-a79b-43bcee5fc7ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:17.596523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.591295    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f640ab64-980a-490e-a79b-43bcee5fc7ca-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:17.716546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.713780    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:17.820408 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:18:17.820369    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.584s"
Oct 31 00:18:17.820744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.820724    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:18:17.820999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.820974    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj" event={"ID":"f640ab64-980a-490e-a79b-43bcee5fc7ca","Type":"ContainerDied","Data":"5e0ef18eef4d91f383a75e0c15437c8731745d31ecbae66c01d4940f76db2202"}
Oct 31 00:18:17.821155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.821136    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e0ef18eef4d91f383a75e0c15437c8731745d31ecbae66c01d4940f76db2202"
Oct 31 00:18:17.936950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.936889    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:18:17.938155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.938113    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:17.958166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.958110    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:17.962638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.962603    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:17.997833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:17.997729    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:18.002708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.002667    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:18.003818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.003778    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:18:18.008806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.008781    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:18:18.011911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.011891    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:18:18.012393 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.012375    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:18:18.012601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.012586    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:18:18.012766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.012752    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:18:18.012927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.012914    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:18:18.154491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.117334    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:18.179489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.145580    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:18.204171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.204138    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:18:18.204535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.204515    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:18:18.212548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.212510    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:18.213113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.213049    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:18.219989 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.219966    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:18:18.220201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.220187    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:18:18.278569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.272320    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:18.285538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.282730    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:18.285538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.283482    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:18:18.696884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:18.696822    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.060545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.028340    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.130736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.048733    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.130736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.063140    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:19.130736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.063381    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.130736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.064010    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.198207    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.248691    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.249690    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.251153    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.251180    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.292594    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:19.389049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.293550    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:19.517497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:19.497004    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded"
Oct 31 00:18:20.047947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.047896    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:18:20.063026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.058502    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:18:20.063026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.060650    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:18:20.078260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.078215    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:20.078549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.078224    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:20.108398 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.099759    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:20.198155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.182783    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:18:20.198155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.188982    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:18:20.207821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.207785    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:20.334455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.333961    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:20.334455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.334230    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:20.334455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.334397    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:20.334795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.334579    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:20.371481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.371428    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca\") pod \"ba29423d-2585-4527-bd0f-ceee7ec96593\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") "
Oct 31 00:18:20.371767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.371730    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp\" (UniqueName: \"kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp\") pod \"ba29423d-2585-4527-bd0f-ceee7ec96593\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") "
Oct 31 00:18:20.408449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.408423    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca\") pod \"ba29423d-2585-4527-bd0f-ceee7ec96593\" (UID: \"ba29423d-2585-4527-bd0f-ceee7ec96593\") "
Oct 31 00:18:20.409577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.409552    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca" pod "ba29423d-2585-4527-bd0f-ceee7ec96593" (UID: "ba29423d-2585-4527-bd0f-ceee7ec96593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:20.495963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.495930    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca" pod "ba29423d-2585-4527-bd0f-ceee7ec96593" (UID: "ba29423d-2585-4527-bd0f-ceee7ec96593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:20.501935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.501434    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:20.501935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.501539    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba29423d-2585-4527-bd0f-ceee7ec96593-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.503282    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp" pod "ba29423d-2585-4527-bd0f-ceee7ec96593" (UID: "ba29423d-2585-4527-bd0f-ceee7ec96593"). InnerVolumeSpecName "kube-api-access-l6scp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.503537    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.503695    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.503947    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.504008    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.504046    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:18:20.504825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.504139    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:18:20.505805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.505436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:18:20.608932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.608846    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6scp\" (UniqueName: \"kubernetes.io/projected/ba29423d-2585-4527-bd0f-ceee7ec96593-kube-api-access-l6scp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:20.920853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.920825    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"
Oct 31 00:18:20.921031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.920845    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb" event={"ID":"ba29423d-2585-4527-bd0f-ceee7ec96593","Type":"ContainerDied","Data":"a0151cf2ab1d2333297f6cd4eefc8202f2a43eac328d667e10993988b5285ea9"}
Oct 31 00:18:20.921031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:20.920926    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0151cf2ab1d2333297f6cd4eefc8202f2a43eac328d667e10993988b5285ea9"
Oct 31 00:18:21.215973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.215904    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"]
Oct 31 00:18:21.216751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216731    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1f171633-a74d-4ec9-a7c3-412672ba9f49" containerName="revoke-keys"
Oct 31 00:18:21.216751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216748    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f171633-a74d-4ec9-a7c3-412672ba9f49" containerName="revoke-keys"
Oct 31 00:18:21.216924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216761    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f640ab64-980a-490e-a79b-43bcee5fc7ca" containerName="revoke-keys"
Oct 31 00:18:21.216924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216768    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f640ab64-980a-490e-a79b-43bcee5fc7ca" containerName="revoke-keys"
Oct 31 00:18:21.216924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216782    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ba29423d-2585-4527-bd0f-ceee7ec96593" containerName="revoke-keys"
Oct 31 00:18:21.216924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216787    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba29423d-2585-4527-bd0f-ceee7ec96593" containerName="revoke-keys"
Oct 31 00:18:21.216924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216924    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f171633-a74d-4ec9-a7c3-412672ba9f49" containerName="revoke-keys"
Oct 31 00:18:21.217109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216933    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f640ab64-980a-490e-a79b-43bcee5fc7ca" containerName="revoke-keys"
Oct 31 00:18:21.217109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.216941    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba29423d-2585-4527-bd0f-ceee7ec96593" containerName="revoke-keys"
Oct 31 00:18:21.221340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.221318    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.227566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.227541    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:18:21.227706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.227649    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:21.232403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.232372    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"]
Oct 31 00:18:21.321410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.321375    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chczn\" (UniqueName: \"kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.321704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.321678    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.321813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.321797    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.425114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.425024    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chczn\" (UniqueName: \"kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.426247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.426222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.427328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.427307    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.428128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.427206    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.444575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.443823    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.447382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.446557    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chczn\" (UniqueName: \"kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn\") pod \"maas-api-revoke-sub-8fc93e3e-grmsx\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.545425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.545327    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:21.859977 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:21.859951    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8640a5d5_91f3_40c4_b062_98dfedb6f1e6.slice/crio-749be5495a1b6b563b5b7c537c66fe17879e967735970ad9ac314f7970e91c1c WatchSource:0}: Error finding container 749be5495a1b6b563b5b7c537c66fe17879e967735970ad9ac314f7970e91c1c: Status 404 returned error can't find the container with id 749be5495a1b6b563b5b7c537c66fe17879e967735970ad9ac314f7970e91c1c
Oct 31 00:18:21.863286 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.863267    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"]
Oct 31 00:18:21.931523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:21.931494    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx" event={"ID":"8640a5d5-91f3-40c4-b062-98dfedb6f1e6","Type":"ContainerStarted","Data":"749be5495a1b6b563b5b7c537c66fe17879e967735970ad9ac314f7970e91c1c"}
Oct 31 00:18:22.899410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:22.899375    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"]
Oct 31 00:18:22.920571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:22.906240    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:22.925830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:22.925797    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"]
Oct 31 00:18:22.948286 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:22.948238    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx" event={"ID":"8640a5d5-91f3-40c4-b062-98dfedb6f1e6","Type":"ContainerStarted","Data":"34746b0aaca066404969dd4a3688c181f16577e8b3ea9213886c00f87b98408d"}
Oct 31 00:18:22.974093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:22.974037    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx" podStartSLOduration=1.974021568 podStartE2EDuration="1.974021568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:22.96865323 +0000 UTC m=+1470.915023946" watchObservedRunningTime="2026-10-31 00:18:22.974021568 +0000 UTC m=+1470.920392284"
Oct 31 00:18:23.055332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.055302    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.055508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.055391    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.055508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.055417    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dn7d\" (UniqueName: \"kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.157894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.157819    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.158058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.157902    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.158058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.157932    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dn7d\" (UniqueName: \"kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.158659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.158638    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.160442 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.160423    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.173083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.173057    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dn7d\" (UniqueName: \"kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d\") pod \"maas-api-revoke-sub-3c82dab9-hvhx5\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.246513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.246452    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:23.538954 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:23.538928    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e9934d_8faf_4370_9c19_01805736a20d.slice/crio-f8e42a1b27e38e7e92b49a19d6e310f83b0de407c1b9c50cc108a3c9d55503d4 WatchSource:0}: Error finding container f8e42a1b27e38e7e92b49a19d6e310f83b0de407c1b9c50cc108a3c9d55503d4: Status 404 returned error can't find the container with id f8e42a1b27e38e7e92b49a19d6e310f83b0de407c1b9c50cc108a3c9d55503d4
Oct 31 00:18:23.545289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.545261    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"]
Oct 31 00:18:23.963770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.963734    5078 generic.go:356] "Generic (PLEG): container finished" podID="8640a5d5-91f3-40c4-b062-98dfedb6f1e6" containerID="34746b0aaca066404969dd4a3688c181f16577e8b3ea9213886c00f87b98408d" exitCode=0
Oct 31 00:18:23.964282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.963838    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx" event={"ID":"8640a5d5-91f3-40c4-b062-98dfedb6f1e6","Type":"ContainerDied","Data":"34746b0aaca066404969dd4a3688c181f16577e8b3ea9213886c00f87b98408d"}
Oct 31 00:18:23.965635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:23.965606    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5" event={"ID":"c6e9934d-8faf-4370-9c19-01805736a20d","Type":"ContainerStarted","Data":"f8e42a1b27e38e7e92b49a19d6e310f83b0de407c1b9c50cc108a3c9d55503d4"}
Oct 31 00:18:24.977345 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:24.977246    5078 generic.go:356] "Generic (PLEG): container finished" podID="c6e9934d-8faf-4370-9c19-01805736a20d" containerID="d88806fbf5ab4a9be8621919dd90b5dfacf139936887eb02e78c3167b2624d22" exitCode=0
Oct 31 00:18:24.977816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:24.977332    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5" event={"ID":"c6e9934d-8faf-4370-9c19-01805736a20d","Type":"ContainerDied","Data":"d88806fbf5ab4a9be8621919dd90b5dfacf139936887eb02e78c3167b2624d22"}
Oct 31 00:18:25.232060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.231965    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:25.416866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.416824    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn\" (UniqueName: \"kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn\") pod \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") "
Oct 31 00:18:25.417058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.417035    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca\") pod \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") "
Oct 31 00:18:25.417181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.417158    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca\") pod \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\" (UID: \"8640a5d5-91f3-40c4-b062-98dfedb6f1e6\") "
Oct 31 00:18:25.417938 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.417908    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca" pod "8640a5d5-91f3-40c4-b062-98dfedb6f1e6" (UID: "8640a5d5-91f3-40c4-b062-98dfedb6f1e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:25.419767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.419746    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca" pod "8640a5d5-91f3-40c4-b062-98dfedb6f1e6" (UID: "8640a5d5-91f3-40c4-b062-98dfedb6f1e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:25.420024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.420005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn" pod "8640a5d5-91f3-40c4-b062-98dfedb6f1e6" (UID: "8640a5d5-91f3-40c4-b062-98dfedb6f1e6"). InnerVolumeSpecName "kube-api-access-chczn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:25.451514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.450762    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"]
Oct 31 00:18:25.452652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.452633    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8640a5d5-91f3-40c4-b062-98dfedb6f1e6" containerName="revoke-keys"
Oct 31 00:18:25.452752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.452655    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8640a5d5-91f3-40c4-b062-98dfedb6f1e6" containerName="revoke-keys"
Oct 31 00:18:25.452927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.452913    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8640a5d5-91f3-40c4-b062-98dfedb6f1e6" containerName="revoke-keys"
Oct 31 00:18:25.462483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.458186    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.471734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.470560    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"]
Oct 31 00:18:25.520247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.520209    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.520437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.520346    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.520643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.520618    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvxj\" (UniqueName: \"kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.520821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.520802    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chczn\" (UniqueName: \"kubernetes.io/projected/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-kube-api-access-chczn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.520882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.520830    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.520882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.520846    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8640a5d5-91f3-40c4-b062-98dfedb6f1e6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.622327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.622294    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.622527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.622455    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.622705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.622687    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvxj\" (UniqueName: \"kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.628601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.624020    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.628601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.626937    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.653487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.650816    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvxj\" (UniqueName: \"kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj\") pod \"maas-api-revoke-sub-40faa617-rq8d2\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.813169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.813062    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:25.989890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.989859    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"
Oct 31 00:18:25.989890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.989882    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx" event={"ID":"8640a5d5-91f3-40c4-b062-98dfedb6f1e6","Type":"ContainerDied","Data":"749be5495a1b6b563b5b7c537c66fe17879e967735970ad9ac314f7970e91c1c"}
Oct 31 00:18:25.990485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:25.989933    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="749be5495a1b6b563b5b7c537c66fe17879e967735970ad9ac314f7970e91c1c"
Oct 31 00:18:26.157990 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:26.157957    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f9f64f_e770_4c36_aad8_6975db123241.slice/crio-d47a7dc86c9d4cc9abdcb3be9d32c443d6d488bb3cb979dbac10a218a364f135 WatchSource:0}: Error finding container d47a7dc86c9d4cc9abdcb3be9d32c443d6d488bb3cb979dbac10a218a364f135: Status 404 returned error can't find the container with id d47a7dc86c9d4cc9abdcb3be9d32c443d6d488bb3cb979dbac10a218a364f135
Oct 31 00:18:26.158154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.158067    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"]
Oct 31 00:18:26.162591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.162569    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:26.183798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.183773    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"]
Oct 31 00:18:26.184543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.184528    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c6e9934d-8faf-4370-9c19-01805736a20d" containerName="revoke-keys"
Oct 31 00:18:26.184608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.184547    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e9934d-8faf-4370-9c19-01805736a20d" containerName="revoke-keys"
Oct 31 00:18:26.184756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.184743    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e9934d-8faf-4370-9c19-01805736a20d" containerName="revoke-keys"
Oct 31 00:18:26.190779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.190754    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.234605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.234575    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca\") pod \"c6e9934d-8faf-4370-9c19-01805736a20d\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") "
Oct 31 00:18:26.234805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.234671    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d\" (UniqueName: \"kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d\") pod \"c6e9934d-8faf-4370-9c19-01805736a20d\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") "
Oct 31 00:18:26.234805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.234798    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca\") pod \"c6e9934d-8faf-4370-9c19-01805736a20d\" (UID: \"c6e9934d-8faf-4370-9c19-01805736a20d\") "
Oct 31 00:18:26.235455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.235431    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5jg\" (UniqueName: \"kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.235658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.235638    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.235801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.235786    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.236344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.236318    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca" pod "c6e9934d-8faf-4370-9c19-01805736a20d" (UID: "c6e9934d-8faf-4370-9c19-01805736a20d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:26.245255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.239332    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d" pod "c6e9934d-8faf-4370-9c19-01805736a20d" (UID: "c6e9934d-8faf-4370-9c19-01805736a20d"). InnerVolumeSpecName "kube-api-access-5dn7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:26.266028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.265996    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca" pod "c6e9934d-8faf-4370-9c19-01805736a20d" (UID: "c6e9934d-8faf-4370-9c19-01805736a20d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:26.274481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.271445    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"]
Oct 31 00:18:26.337558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.337527    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.337697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.337673    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.337897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.337877    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5jg\" (UniqueName: \"kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.338116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.338098    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dn7d\" (UniqueName: \"kubernetes.io/projected/c6e9934d-8faf-4370-9c19-01805736a20d-kube-api-access-5dn7d\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.338167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.338123    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.338167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.338139    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e9934d-8faf-4370-9c19-01805736a20d-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.338630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.338609    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.340669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.340647    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.351742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.351722    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5jg\" (UniqueName: \"kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg\") pod \"maas-api-revoke-sub-218254ba-gdbx8\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.604443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.604411    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:26.910795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:26.910758    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"]
Oct 31 00:18:27.002098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.002057    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8" event={"ID":"15489863-c603-4bd2-8d51-7e90cdf15244","Type":"ContainerStarted","Data":"5629c9f07b9cc82cfdc6bf60e19fea8b982ba79e16803267cb43218e9d8f99c5"}
Oct 31 00:18:27.004406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.004378    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2" event={"ID":"b2f9f64f-e770-4c36-aad8-6975db123241","Type":"ContainerStarted","Data":"626ec469a2526fad061c53303b8e6c63b1ac5d8aeb4b4a36d00980e6f12c9f2a"}
Oct 31 00:18:27.004608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.004431    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2" event={"ID":"b2f9f64f-e770-4c36-aad8-6975db123241","Type":"ContainerStarted","Data":"d47a7dc86c9d4cc9abdcb3be9d32c443d6d488bb3cb979dbac10a218a364f135"}
Oct 31 00:18:27.006344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.006258    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5" event={"ID":"c6e9934d-8faf-4370-9c19-01805736a20d","Type":"ContainerDied","Data":"f8e42a1b27e38e7e92b49a19d6e310f83b0de407c1b9c50cc108a3c9d55503d4"}
Oct 31 00:18:27.006344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.006299    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e42a1b27e38e7e92b49a19d6e310f83b0de407c1b9c50cc108a3c9d55503d4"
Oct 31 00:18:27.006344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.006308    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"
Oct 31 00:18:27.035686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.035640    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2" podStartSLOduration=2.035626091 podStartE2EDuration="2.035626091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:27.031710882 +0000 UTC m=+1474.978081600" watchObservedRunningTime="2026-10-31 00:18:27.035626091 +0000 UTC m=+1474.981996850"
Oct 31 00:18:27.515660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.515582    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"]
Oct 31 00:18:27.520631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.520607    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.532387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.532359    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"]
Oct 31 00:18:27.559282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.559255    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.559436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.559322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.559515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.559488    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrw9\" (UniqueName: \"kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.660731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.660700    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrw9\" (UniqueName: \"kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.660888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.660845    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.660935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.660889    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.661568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.661546    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.663504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.663481    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.675121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.675097    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrw9\" (UniqueName: \"kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9\") pod \"maas-api-revoke-sub-eb0a1b16-8jt8c\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:27.836886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:27.836752    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:28.023484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:28.023103    5078 generic.go:356] "Generic (PLEG): container finished" podID="15489863-c603-4bd2-8d51-7e90cdf15244" containerID="b631f6e8ee0dc568404e57b5abbc9971f629af7c9ee0fd7ea2f0f6f8902124cf" exitCode=0
Oct 31 00:18:28.023484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:28.023213    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8" event={"ID":"15489863-c603-4bd2-8d51-7e90cdf15244","Type":"ContainerDied","Data":"b631f6e8ee0dc568404e57b5abbc9971f629af7c9ee0fd7ea2f0f6f8902124cf"}
Oct 31 00:18:28.026186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:28.026152    5078 generic.go:356] "Generic (PLEG): container finished" podID="b2f9f64f-e770-4c36-aad8-6975db123241" containerID="626ec469a2526fad061c53303b8e6c63b1ac5d8aeb4b4a36d00980e6f12c9f2a" exitCode=0
Oct 31 00:18:28.026321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:28.026231    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2" event={"ID":"b2f9f64f-e770-4c36-aad8-6975db123241","Type":"ContainerDied","Data":"626ec469a2526fad061c53303b8e6c63b1ac5d8aeb4b4a36d00980e6f12c9f2a"}
Oct 31 00:18:28.165983 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:28.165955    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4d04b6_98ea_41cd_8ecd_77ca78966a01.slice/crio-fae701b93b0a676c07c4f28f2fce8299048151437e31761ba865048ceec35f50 WatchSource:0}: Error finding container fae701b93b0a676c07c4f28f2fce8299048151437e31761ba865048ceec35f50: Status 404 returned error can't find the container with id fae701b93b0a676c07c4f28f2fce8299048151437e31761ba865048ceec35f50
Oct 31 00:18:28.168036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:28.168011    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"]
Oct 31 00:18:29.036738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.036695    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c" event={"ID":"8a4d04b6-98ea-41cd-8ecd-77ca78966a01","Type":"ContainerStarted","Data":"44abcaecb39656baa0433eba6e3a5e1777e94230e6bee169ba35528573294891"}
Oct 31 00:18:29.037245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.036809    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c" event={"ID":"8a4d04b6-98ea-41cd-8ecd-77ca78966a01","Type":"ContainerStarted","Data":"fae701b93b0a676c07c4f28f2fce8299048151437e31761ba865048ceec35f50"}
Oct 31 00:18:29.067400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.067335    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c" podStartSLOduration=2.067316056 podStartE2EDuration="2.067316056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:29.060484308 +0000 UTC m=+1477.006855021" watchObservedRunningTime="2026-10-31 00:18:29.067316056 +0000 UTC m=+1477.013686779"
Oct 31 00:18:29.282411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.282379    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:29.327366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.325214    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca\") pod \"b2f9f64f-e770-4c36-aad8-6975db123241\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") "
Oct 31 00:18:29.327366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.325274    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj\" (UniqueName: \"kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj\") pod \"b2f9f64f-e770-4c36-aad8-6975db123241\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") "
Oct 31 00:18:29.327366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.326428    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca\") pod \"b2f9f64f-e770-4c36-aad8-6975db123241\" (UID: \"b2f9f64f-e770-4c36-aad8-6975db123241\") "
Oct 31 00:18:29.327366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.326928    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca" pod "b2f9f64f-e770-4c36-aad8-6975db123241" (UID: "b2f9f64f-e770-4c36-aad8-6975db123241"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:29.327366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.327336    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:29.339022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.336691    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj" pod "b2f9f64f-e770-4c36-aad8-6975db123241" (UID: "b2f9f64f-e770-4c36-aad8-6975db123241"). InnerVolumeSpecName "kube-api-access-gmvxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:29.339022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.336858    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca" pod "b2f9f64f-e770-4c36-aad8-6975db123241" (UID: "b2f9f64f-e770-4c36-aad8-6975db123241"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:29.396525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.396497    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:29.428478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.428430    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca\") pod \"15489863-c603-4bd2-8d51-7e90cdf15244\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") "
Oct 31 00:18:29.428644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.428630    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg\" (UniqueName: \"kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg\") pod \"15489863-c603-4bd2-8d51-7e90cdf15244\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") "
Oct 31 00:18:29.428762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.428748    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca\") pod \"15489863-c603-4bd2-8d51-7e90cdf15244\" (UID: \"15489863-c603-4bd2-8d51-7e90cdf15244\") "
Oct 31 00:18:29.429714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.429690    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f9f64f-e770-4c36-aad8-6975db123241-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:29.429848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.429717    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvxj\" (UniqueName: \"kubernetes.io/projected/b2f9f64f-e770-4c36-aad8-6975db123241-kube-api-access-gmvxj\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:29.431347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.431324    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca" pod "15489863-c603-4bd2-8d51-7e90cdf15244" (UID: "15489863-c603-4bd2-8d51-7e90cdf15244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:29.432070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.432044    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg" pod "15489863-c603-4bd2-8d51-7e90cdf15244" (UID: "15489863-c603-4bd2-8d51-7e90cdf15244"). InnerVolumeSpecName "kube-api-access-hk5jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:29.432845 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.432821    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca" pod "15489863-c603-4bd2-8d51-7e90cdf15244" (UID: "15489863-c603-4bd2-8d51-7e90cdf15244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:29.530928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.530897    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:29.530928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.530927    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15489863-c603-4bd2-8d51-7e90cdf15244-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:29.530928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:29.530936    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk5jg\" (UniqueName: \"kubernetes.io/projected/15489863-c603-4bd2-8d51-7e90cdf15244-kube-api-access-hk5jg\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:30.048955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.048922    5078 generic.go:356] "Generic (PLEG): container finished" podID="8a4d04b6-98ea-41cd-8ecd-77ca78966a01" containerID="44abcaecb39656baa0433eba6e3a5e1777e94230e6bee169ba35528573294891" exitCode=0
Oct 31 00:18:30.049436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.048995    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c" event={"ID":"8a4d04b6-98ea-41cd-8ecd-77ca78966a01","Type":"ContainerDied","Data":"44abcaecb39656baa0433eba6e3a5e1777e94230e6bee169ba35528573294891"}
Oct 31 00:18:30.050621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.050602    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"
Oct 31 00:18:30.050751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.050618    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8" event={"ID":"15489863-c603-4bd2-8d51-7e90cdf15244","Type":"ContainerDied","Data":"5629c9f07b9cc82cfdc6bf60e19fea8b982ba79e16803267cb43218e9d8f99c5"}
Oct 31 00:18:30.050751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.050659    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5629c9f07b9cc82cfdc6bf60e19fea8b982ba79e16803267cb43218e9d8f99c5"
Oct 31 00:18:30.053046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.053028    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"
Oct 31 00:18:30.053046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.053040    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2" event={"ID":"b2f9f64f-e770-4c36-aad8-6975db123241","Type":"ContainerDied","Data":"d47a7dc86c9d4cc9abdcb3be9d32c443d6d488bb3cb979dbac10a218a364f135"}
Oct 31 00:18:30.053164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.053068    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47a7dc86c9d4cc9abdcb3be9d32c443d6d488bb3cb979dbac10a218a364f135"
Oct 31 00:18:30.353338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.353252    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"]
Oct 31 00:18:30.354032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.354017    5078 cpu_manager.go:414] "containerMap: removing container" podUID="15489863-c603-4bd2-8d51-7e90cdf15244" containerName="revoke-keys"
Oct 31 00:18:30.354101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.354033    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15489863-c603-4bd2-8d51-7e90cdf15244" containerName="revoke-keys"
Oct 31 00:18:30.354101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.354045    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b2f9f64f-e770-4c36-aad8-6975db123241" containerName="revoke-keys"
Oct 31 00:18:30.354101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.354050    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f9f64f-e770-4c36-aad8-6975db123241" containerName="revoke-keys"
Oct 31 00:18:30.354206 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.354197    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="15489863-c603-4bd2-8d51-7e90cdf15244" containerName="revoke-keys"
Oct 31 00:18:30.354238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.354209    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f9f64f-e770-4c36-aad8-6975db123241" containerName="revoke-keys"
Oct 31 00:18:30.358260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.358241    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.368060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.368038    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"]
Oct 31 00:18:30.445956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.445921    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.446107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.446068    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zcb\" (UniqueName: \"kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.446266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.446234    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.548069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.548034    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zcb\" (UniqueName: \"kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.548227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.548143    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.548318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.548280    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.548979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.548936    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.550939 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.550917    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.562007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.561981    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zcb\" (UniqueName: \"kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb\") pod \"maas-api-revoke-sub-972f7613-dhn5n\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:30.690317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:30.689737    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:31.010689 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:31.010656    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546fbce6_4e3d_4829_bc58_cb72086abd10.slice/crio-463d9950dcde7402d1f05ff61164d0b061ebc8e3651ffb464b25303ef06f2fd3 WatchSource:0}: Error finding container 463d9950dcde7402d1f05ff61164d0b061ebc8e3651ffb464b25303ef06f2fd3: Status 404 returned error can't find the container with id 463d9950dcde7402d1f05ff61164d0b061ebc8e3651ffb464b25303ef06f2fd3
Oct 31 00:18:31.011448 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.011404    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"]
Oct 31 00:18:31.063095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.063060    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n" event={"ID":"546fbce6-4e3d-4829-bc58-cb72086abd10","Type":"ContainerStarted","Data":"463d9950dcde7402d1f05ff61164d0b061ebc8e3651ffb464b25303ef06f2fd3"}
Oct 31 00:18:31.236090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.236061    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:31.404700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.404662    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca\") pod \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") "
Oct 31 00:18:31.404893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.404818    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9\" (UniqueName: \"kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9\") pod \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") "
Oct 31 00:18:31.404893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.404879    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca\") pod \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\" (UID: \"8a4d04b6-98ea-41cd-8ecd-77ca78966a01\") "
Oct 31 00:18:31.405098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.405069    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca" pod "8a4d04b6-98ea-41cd-8ecd-77ca78966a01" (UID: "8a4d04b6-98ea-41cd-8ecd-77ca78966a01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:31.405682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.405664    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:31.407235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.407208    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca" pod "8a4d04b6-98ea-41cd-8ecd-77ca78966a01" (UID: "8a4d04b6-98ea-41cd-8ecd-77ca78966a01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:31.407550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.407528    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9" pod "8a4d04b6-98ea-41cd-8ecd-77ca78966a01" (UID: "8a4d04b6-98ea-41cd-8ecd-77ca78966a01"). InnerVolumeSpecName "kube-api-access-pbrw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:31.507232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.507194    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrw9\" (UniqueName: \"kubernetes.io/projected/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-kube-api-access-pbrw9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:31.507232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:31.507224    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4d04b6-98ea-41cd-8ecd-77ca78966a01-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:32.077103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:32.077075    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"
Oct 31 00:18:32.077103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:32.077068    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c" event={"ID":"8a4d04b6-98ea-41cd-8ecd-77ca78966a01","Type":"ContainerDied","Data":"fae701b93b0a676c07c4f28f2fce8299048151437e31761ba865048ceec35f50"}
Oct 31 00:18:32.077663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:32.077169    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae701b93b0a676c07c4f28f2fce8299048151437e31761ba865048ceec35f50"
Oct 31 00:18:32.079269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:32.079237    5078 generic.go:356] "Generic (PLEG): container finished" podID="546fbce6-4e3d-4829-bc58-cb72086abd10" containerID="944483f7a04a457707080c31f182ea000b8bc260e782a362c1dc70739bdfb073" exitCode=0
Oct 31 00:18:32.079403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:32.079333    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n" event={"ID":"546fbce6-4e3d-4829-bc58-cb72086abd10","Type":"ContainerDied","Data":"944483f7a04a457707080c31f182ea000b8bc260e782a362c1dc70739bdfb073"}
Oct 31 00:18:33.196336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.195981    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"]
Oct 31 00:18:33.197263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.197243    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8a4d04b6-98ea-41cd-8ecd-77ca78966a01" containerName="revoke-keys"
Oct 31 00:18:33.197419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.197266    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4d04b6-98ea-41cd-8ecd-77ca78966a01" containerName="revoke-keys"
Oct 31 00:18:33.197563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.197483    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4d04b6-98ea-41cd-8ecd-77ca78966a01" containerName="revoke-keys"
Oct 31 00:18:33.215718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.215692    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.233514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.229161    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"]
Oct 31 00:18:33.364044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.363955    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wj58\" (UniqueName: \"kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.364044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.364013    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.364263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.364213    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.382520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.382497    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:33.465581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.465544    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca\") pod \"546fbce6-4e3d-4829-bc58-cb72086abd10\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") "
Oct 31 00:18:33.465747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.465641    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb\" (UniqueName: \"kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb\") pod \"546fbce6-4e3d-4829-bc58-cb72086abd10\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") "
Oct 31 00:18:33.465836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.465806    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca\") pod \"546fbce6-4e3d-4829-bc58-cb72086abd10\" (UID: \"546fbce6-4e3d-4829-bc58-cb72086abd10\") "
Oct 31 00:18:33.466262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.466234    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca" pod "546fbce6-4e3d-4829-bc58-cb72086abd10" (UID: "546fbce6-4e3d-4829-bc58-cb72086abd10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:33.466446 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.466422    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wj58\" (UniqueName: \"kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.466561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.466487    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.466696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.466677    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.466997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.466966    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:33.467304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.467279    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.469558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.469521    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca" pod "546fbce6-4e3d-4829-bc58-cb72086abd10" (UID: "546fbce6-4e3d-4829-bc58-cb72086abd10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:33.470512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.469864    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb" pod "546fbce6-4e3d-4829-bc58-cb72086abd10" (UID: "546fbce6-4e3d-4829-bc58-cb72086abd10"). InnerVolumeSpecName "kube-api-access-z4zcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:33.470512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.470289    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.480347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.480314    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wj58\" (UniqueName: \"kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58\") pod \"maas-api-revoke-sub-0e430b93-jw6mr\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.536340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.536316    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:33.568440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.568416    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546fbce6-4e3d-4829-bc58-cb72086abd10-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:33.568574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.568447    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zcb\" (UniqueName: \"kubernetes.io/projected/546fbce6-4e3d-4829-bc58-cb72086abd10-kube-api-access-z4zcb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:33.835513 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:33.835482    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ef0771_1298_435d_82f8_7716d0feb818.slice/crio-2b75d2b66e7cb7691a1aaa3344d82b784c428a1600f965ac0722b9e358c2abe5 WatchSource:0}: Error finding container 2b75d2b66e7cb7691a1aaa3344d82b784c428a1600f965ac0722b9e358c2abe5: Status 404 returned error can't find the container with id 2b75d2b66e7cb7691a1aaa3344d82b784c428a1600f965ac0722b9e358c2abe5
Oct 31 00:18:33.838500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:33.838439    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"]
Oct 31 00:18:34.102929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:34.102893    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n" event={"ID":"546fbce6-4e3d-4829-bc58-cb72086abd10","Type":"ContainerDied","Data":"463d9950dcde7402d1f05ff61164d0b061ebc8e3651ffb464b25303ef06f2fd3"}
Oct 31 00:18:34.103081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:34.102949    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="463d9950dcde7402d1f05ff61164d0b061ebc8e3651ffb464b25303ef06f2fd3"
Oct 31 00:18:34.103081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:34.103014    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"
Oct 31 00:18:34.107643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:34.107612    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr" event={"ID":"91ef0771-1298-435d-82f8-7716d0feb818","Type":"ContainerStarted","Data":"2b75d2b66e7cb7691a1aaa3344d82b784c428a1600f965ac0722b9e358c2abe5"}
Oct 31 00:18:35.187486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:35.177974    5078 generic.go:356] "Generic (PLEG): container finished" podID="91ef0771-1298-435d-82f8-7716d0feb818" containerID="7b9f2057ec0aebfea8a938a801494002f3a0f5087fadb39798288743531de0b2" exitCode=0
Oct 31 00:18:35.187486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:35.178100    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr" event={"ID":"91ef0771-1298-435d-82f8-7716d0feb818","Type":"ContainerDied","Data":"7b9f2057ec0aebfea8a938a801494002f3a0f5087fadb39798288743531de0b2"}
Oct 31 00:18:35.941976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:35.941841    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.073612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.073566    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.074651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.074621    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.077118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.077092    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.094549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.080539    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.332895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.332778    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:36.333833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.333804    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:36.334245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.334220    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:36.335411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.335384    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.396489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.373578    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.400521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.400401    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:36.422669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.373599    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.443689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:36.443569    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:36.975669 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:18:36.975631    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:18:37.033473 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.033423    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"]
Oct 31 00:18:37.040498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.034935    5078 cpu_manager.go:414] "containerMap: removing container" podUID="546fbce6-4e3d-4829-bc58-cb72086abd10" containerName="revoke-keys"
Oct 31 00:18:37.040694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.040672    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546fbce6-4e3d-4829-bc58-cb72086abd10" containerName="revoke-keys"
Oct 31 00:18:37.041065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.041045    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="546fbce6-4e3d-4829-bc58-cb72086abd10" containerName="revoke-keys"
Oct 31 00:18:37.088267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.088162    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:18:37.129080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.129035    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:37.441552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.387995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:37.441552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.388417    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:37.441552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.388996    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.442366    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.483575    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.526920    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.526976    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.527014    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.527049    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.527084    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.527119    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.527152    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:18:37.529993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.528226    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:37.585855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.585803    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Oct 31 00:18:37.618316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.589674    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:37.618608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.616608    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:37.771548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.677729    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:37.771548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.729329    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:37.825493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.801061    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:18:37.825847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.825823    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:18:37.863897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.863867    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:18:37.922533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.919176    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfhx\" (UniqueName: \"kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:37.922533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.919644    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:37.922533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:37.919698    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:38.424737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.424612    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:38.460707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.460660    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:38.477903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.428536    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:38.540490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.428594    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:38.540490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.530225    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:18:38.584490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.559785    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:38.622259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.622210    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:38.747919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:38.730635    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.106204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.106098    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.107066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.107035    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.111620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.107623    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.145699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.107667    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.232703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.107688    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.268725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.107719    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 31 00:18:39.268725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.156454    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:39.268725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.259708    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:39.463329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.463273    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:39.520494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.484025    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:39.520494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.484373    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.520494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.484673    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.520494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.518607    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:39.616489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.613583    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:39.620652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.620543    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:18:39.657490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.632617    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:39.683487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.644738    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:39.743488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.631692    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfhx\" (UniqueName: \"kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:39.759517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.759381    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfhx\" (UniqueName: \"kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:39.899490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.894545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:39.899490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.897698    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9285ec5-4xqtc\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:39.990485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:39.951753    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:40.285487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.281914    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:40.378148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.339571    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:18:40.378394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.371143    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.437490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.431623    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:40.465044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.464974    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.490019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.489920    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:40.682490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.572634    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.699499 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:18:40.573374    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.862s"
Oct 31 00:18:40.699499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.698875    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:18:40.699499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.698913    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"]
Oct 31 00:18:40.699499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.698988    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:18:40.794571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.767562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:40.794571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.793809    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.794571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.794239    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.794571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.794563    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.795994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.795965    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.796486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.796441    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:18:40.928821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.928718    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 31 00:18:40.998535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:40.978229    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:18:41.149490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.147994    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:41.258571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.247080    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca\") pod \"91ef0771-1298-435d-82f8-7716d0feb818\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") "
Oct 31 00:18:41.258571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.247252    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58\" (UniqueName: \"kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58\") pod \"91ef0771-1298-435d-82f8-7716d0feb818\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") "
Oct 31 00:18:41.258571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.247372    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca\") pod \"91ef0771-1298-435d-82f8-7716d0feb818\" (UID: \"91ef0771-1298-435d-82f8-7716d0feb818\") "
Oct 31 00:18:41.431382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.427560    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.431659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.431624    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58" pod "91ef0771-1298-435d-82f8-7716d0feb818" (UID: "91ef0771-1298-435d-82f8-7716d0feb818"). InnerVolumeSpecName "kube-api-access-4wj58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:41.440770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.440733    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca" pod "91ef0771-1298-435d-82f8-7716d0feb818" (UID: "91ef0771-1298-435d-82f8-7716d0feb818"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:41.466490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.441704    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.467520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.457611    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.471554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.468150    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.481234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.481202    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca" pod "91ef0771-1298-435d-82f8-7716d0feb818" (UID: "91ef0771-1298-435d-82f8-7716d0feb818"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:41.634652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.588853    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:41.670508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.665702    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wj58\" (UniqueName: \"kubernetes.io/projected/91ef0771-1298-435d-82f8-7716d0feb818-kube-api-access-4wj58\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:41.698491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.675502    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:18:41.732669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.691455    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:41.773851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.773790    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.777067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.777033    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ef0771-1298-435d-82f8-7716d0feb818-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:41.814692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.814632    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.818769 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.818731    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.846394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.845881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:41.846394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.846275    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:41.897269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:41.897164    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:42.032026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.031949    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:18:42.142332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.083700    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:18:42.142767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.100613    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:42.142977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.142933    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:42.143599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.143579    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"]
Oct 31 00:18:42.285550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.284813    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:42.285550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.284875    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:42.429422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.429388    5078 cpu_manager.go:414] "containerMap: removing container" podUID="91ef0771-1298-435d-82f8-7716d0feb818" containerName="revoke-keys"
Oct 31 00:18:42.454542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.454504    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ef0771-1298-435d-82f8-7716d0feb818" containerName="revoke-keys"
Oct 31 00:18:42.455130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.455104    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ef0771-1298-435d-82f8-7716d0feb818" containerName="revoke-keys"
Oct 31 00:18:42.475489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.457982    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:42.476135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.476092    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:42.476535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.431946    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:18:42.479654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.479616    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:42.480147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.480117    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:42.492493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.480559    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:42.613059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.576453    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"]
Oct 31 00:18:42.673181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.663626    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:42.798496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.796626    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:42.811608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.811542    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:42.911491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.908225    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:43.399593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.386517    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:43.399593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.941513    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:43.412369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.915585    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.515873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.941545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:43.534862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.960760    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:18:43.537541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:42.960900    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.676562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.048879    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.677802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.024443    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:43.678044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.677991    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.678249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.539563    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.744630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.052054    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.766575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.416649    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbf4d\" (UniqueName: \"kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:43.767126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.050332    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:43.771648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.771609    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:43.771920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.402107    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:43.775620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.775558    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.018493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:43.957526    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.139597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.045710    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:44.139907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.139867    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.156496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.149315    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.164156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.164114    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.164417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.164394    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.164666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.164647    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbf4d\" (UniqueName: \"kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.216336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.216301    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.252788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.252754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.339491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.314120    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:44.467658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.467612    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.397674    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.569819    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.510506    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.510425    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.510574    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.540609    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.605679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.540691    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:44.638997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.638944    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:44.755482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.707656    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.755482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.719925    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:44.755482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.724212    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbf4d\" (UniqueName: \"kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d\") pod \"maas-api-revoke-sub-4ab88dc0-nww8b\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.755482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.746763    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.755482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.747113    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.755482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.748399    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.963641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.892846    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:44.963641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.892933    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:44.963641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.894215    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:44.963641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:44.900170    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 31 00:18:45.194189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.192785    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:45.194189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.192851    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:45.853569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.841489    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:18:45.853569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.846186    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"
Oct 31 00:18:45.853569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.848412    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:18:45.853569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.849100    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:18:45.853569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.849486    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:18:45.856305 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.856275    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr" event={"ID":"91ef0771-1298-435d-82f8-7716d0feb818","Type":"ContainerDied","Data":"2b75d2b66e7cb7691a1aaa3344d82b784c428a1600f965ac0722b9e358c2abe5"}
Oct 31 00:18:45.856438 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.856424    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b75d2b66e7cb7691a1aaa3344d82b784c428a1600f965ac0722b9e358c2abe5"
Oct 31 00:18:45.920696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.919157    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:18:45.943618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.943576    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 31 00:18:45.960515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.946671    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:18:45.960515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.948238    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.003481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.979172    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:46.013624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.013588    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.015032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.980531    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:46.077478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.057617    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.077478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.058042    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.077478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.058361    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.077478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.058735    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.098484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.094111    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:46.115874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:45.898582    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:18:46.158012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.148016    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:18:46.158012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.151032    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 00:18:46.175494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.163788    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:18:46.175494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.166882    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:18:46.175494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.169266    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:18:46.175494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.169311    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:18:46.175494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.171028    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:18:46.175494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.173768    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:18:46.179700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.178723    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:18:46.187077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.187056    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:18:46.188811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.187899    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:18:46.233577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.233553    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:18:46.233670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.233616    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:18:46.301577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.301544    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"]
Oct 31 00:18:46.320692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.320669    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"]
Oct 31 00:18:46.327606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.327585    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.328721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.328696    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"]
Oct 31 00:18:46.440644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.440601    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.440835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.440672    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.440999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.440977    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6l8x\" (UniqueName: \"kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.453657 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:46.453545    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497275d4_4d81_4d69_8c6b_b3daec08ffc0.slice/crio-457a03a4ea8d0d669696fbbec148d91ae9f9996082be2c00091be791eced9287 WatchSource:0}: Error finding container 457a03a4ea8d0d669696fbbec148d91ae9f9996082be2c00091be791eced9287: Status 404 returned error can't find the container with id 457a03a4ea8d0d669696fbbec148d91ae9f9996082be2c00091be791eced9287
Oct 31 00:18:46.454645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.453931    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"]
Oct 31 00:18:46.543007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.542968    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.543312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.543276    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6l8x\" (UniqueName: \"kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.543543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.543524    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.545407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.545378    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.546840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.546811    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.560210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.560188    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6l8x\" (UniqueName: \"kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x\") pod \"maas-api-revoke-sub-befaf1e5-q4hm7\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.649649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.649620    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:46.857508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.857452    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc" event={"ID":"5722e205-4aab-4446-b1ea-a244b5f3da75","Type":"ContainerStarted","Data":"7d553752383a539f1c28d159dbfdc103cb83146244c102f83b92bbffa217e632"}
Oct 31 00:18:46.858003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.857522    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc" event={"ID":"5722e205-4aab-4446-b1ea-a244b5f3da75","Type":"ContainerStarted","Data":"858ab554c3c23dd434d1fff68bb006e40fa52aa146e10484eaa179d805aa2f45"}
Oct 31 00:18:46.859728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.859699    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b" event={"ID":"497275d4-4d81-4d69-8c6b-b3daec08ffc0","Type":"ContainerStarted","Data":"457a03a4ea8d0d669696fbbec148d91ae9f9996082be2c00091be791eced9287"}
Oct 31 00:18:46.891192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:46.891140    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc" podStartSLOduration=10.891122447 podStartE2EDuration="10.891122447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:46.884808617 +0000 UTC m=+1494.831179364" watchObservedRunningTime="2026-10-31 00:18:46.891122447 +0000 UTC m=+1494.837493165"
Oct 31 00:18:47.019527 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:47.019495    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f26466_4fc1_4b05_bda6_5b0cc29d4eb6.slice/crio-fcf63073a5ab748224126256937d580a629df2dccf20a09dd98d64e839c0f453 WatchSource:0}: Error finding container fcf63073a5ab748224126256937d580a629df2dccf20a09dd98d64e839c0f453: Status 404 returned error can't find the container with id fcf63073a5ab748224126256937d580a629df2dccf20a09dd98d64e839c0f453
Oct 31 00:18:47.024285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.024257    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"]
Oct 31 00:18:47.870717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.870636    5078 generic.go:356] "Generic (PLEG): container finished" podID="497275d4-4d81-4d69-8c6b-b3daec08ffc0" containerID="39dcb51ffa769be4f7a9b627508c2bb3e4c50aeb1e1f76a0faabd78880d3ae03" exitCode=0
Oct 31 00:18:47.871124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.870721    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b" event={"ID":"497275d4-4d81-4d69-8c6b-b3daec08ffc0","Type":"ContainerDied","Data":"39dcb51ffa769be4f7a9b627508c2bb3e4c50aeb1e1f76a0faabd78880d3ae03"}
Oct 31 00:18:47.873034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.873006    5078 generic.go:356] "Generic (PLEG): container finished" podID="5722e205-4aab-4446-b1ea-a244b5f3da75" containerID="7d553752383a539f1c28d159dbfdc103cb83146244c102f83b92bbffa217e632" exitCode=0
Oct 31 00:18:47.873392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.873372    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc" event={"ID":"5722e205-4aab-4446-b1ea-a244b5f3da75","Type":"ContainerDied","Data":"7d553752383a539f1c28d159dbfdc103cb83146244c102f83b92bbffa217e632"}
Oct 31 00:18:47.875335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.875310    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7" event={"ID":"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6","Type":"ContainerStarted","Data":"6634a064e89192fdf22d8d3b73866b99f86cf0c8702895d322ca01483a377445"}
Oct 31 00:18:47.875430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.875386    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7" event={"ID":"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6","Type":"ContainerStarted","Data":"fcf63073a5ab748224126256937d580a629df2dccf20a09dd98d64e839c0f453"}
Oct 31 00:18:47.918058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:47.918008    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7" podStartSLOduration=3.917992013 podStartE2EDuration="3.917992013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:47.913411697 +0000 UTC m=+1495.859782417" watchObservedRunningTime="2026-10-31 00:18:47.917992013 +0000 UTC m=+1495.864362729"
Oct 31 00:18:48.888113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:48.888074    5078 generic.go:356] "Generic (PLEG): container finished" podID="19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" containerID="6634a064e89192fdf22d8d3b73866b99f86cf0c8702895d322ca01483a377445" exitCode=0
Oct 31 00:18:48.888693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:48.888159    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7" event={"ID":"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6","Type":"ContainerDied","Data":"6634a064e89192fdf22d8d3b73866b99f86cf0c8702895d322ca01483a377445"}
Oct 31 00:18:49.175559 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.175528    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:49.180090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.180061    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.210911    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx\" (UniqueName: \"kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx\") pod \"5722e205-4aab-4446-b1ea-a244b5f3da75\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") "
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.211001    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca\") pod \"5722e205-4aab-4446-b1ea-a244b5f3da75\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") "
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.211050    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca\") pod \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") "
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.211227    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d\" (UniqueName: \"kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d\") pod \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") "
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.211345    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca\") pod \"5722e205-4aab-4446-b1ea-a244b5f3da75\" (UID: \"5722e205-4aab-4446-b1ea-a244b5f3da75\") "
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.211550    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca\") pod \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\" (UID: \"497275d4-4d81-4d69-8c6b-b3daec08ffc0\") "
Oct 31 00:18:49.215113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.213851    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca" pod "497275d4-4d81-4d69-8c6b-b3daec08ffc0" (UID: "497275d4-4d81-4d69-8c6b-b3daec08ffc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:49.217197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.216865    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca" pod "5722e205-4aab-4446-b1ea-a244b5f3da75" (UID: "5722e205-4aab-4446-b1ea-a244b5f3da75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:49.224499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.222338    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca" pod "497275d4-4d81-4d69-8c6b-b3daec08ffc0" (UID: "497275d4-4d81-4d69-8c6b-b3daec08ffc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:49.224499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.222650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d" pod "497275d4-4d81-4d69-8c6b-b3daec08ffc0" (UID: "497275d4-4d81-4d69-8c6b-b3daec08ffc0"). InnerVolumeSpecName "kube-api-access-bbf4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:49.226735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.224801    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca" pod "5722e205-4aab-4446-b1ea-a244b5f3da75" (UID: "5722e205-4aab-4446-b1ea-a244b5f3da75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:49.236935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.236913    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx" pod "5722e205-4aab-4446-b1ea-a244b5f3da75" (UID: "5722e205-4aab-4446-b1ea-a244b5f3da75"). InnerVolumeSpecName "kube-api-access-fwfhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:49.313892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.313860    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbf4d\" (UniqueName: \"kubernetes.io/projected/497275d4-4d81-4d69-8c6b-b3daec08ffc0-kube-api-access-bbf4d\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:49.314040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.313895    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:49.314040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.313913    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:49.314040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.313928    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfhx\" (UniqueName: \"kubernetes.io/projected/5722e205-4aab-4446-b1ea-a244b5f3da75-kube-api-access-fwfhx\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:49.314040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.313943    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5722e205-4aab-4446-b1ea-a244b5f3da75-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:49.314040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.313956    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497275d4-4d81-4d69-8c6b-b3daec08ffc0-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:49.901680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.901643    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"
Oct 31 00:18:49.902159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.901676    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc" event={"ID":"5722e205-4aab-4446-b1ea-a244b5f3da75","Type":"ContainerDied","Data":"858ab554c3c23dd434d1fff68bb006e40fa52aa146e10484eaa179d805aa2f45"}
Oct 31 00:18:49.902159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.901736    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858ab554c3c23dd434d1fff68bb006e40fa52aa146e10484eaa179d805aa2f45"
Oct 31 00:18:49.904496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.904272    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"
Oct 31 00:18:49.904496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.904302    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b" event={"ID":"497275d4-4d81-4d69-8c6b-b3daec08ffc0","Type":"ContainerDied","Data":"457a03a4ea8d0d669696fbbec148d91ae9f9996082be2c00091be791eced9287"}
Oct 31 00:18:49.904496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:49.904358    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457a03a4ea8d0d669696fbbec148d91ae9f9996082be2c00091be791eced9287"
Oct 31 00:18:50.071118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.071096    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:50.125855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.125833    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca\") pod \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") "
Oct 31 00:18:50.126087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.126069    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x\" (UniqueName: \"kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x\") pod \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") "
Oct 31 00:18:50.126179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.126163    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca\") pod \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\" (UID: \"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6\") "
Oct 31 00:18:50.126518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.126496    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca" pod "19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" (UID: "19f26466-4fc1-4b05-bda6-5b0cc29d4eb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:50.126891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.126873    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:50.128093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.128069    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca" pod "19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" (UID: "19f26466-4fc1-4b05-bda6-5b0cc29d4eb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:50.128385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.128365    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x" pod "19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" (UID: "19f26466-4fc1-4b05-bda6-5b0cc29d4eb6"). InnerVolumeSpecName "kube-api-access-r6l8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:50.228848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.228815    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:50.228848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.228839    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6l8x\" (UniqueName: \"kubernetes.io/projected/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6-kube-api-access-r6l8x\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:50.918257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.918224    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"
Oct 31 00:18:50.918719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.918224    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7" event={"ID":"19f26466-4fc1-4b05-bda6-5b0cc29d4eb6","Type":"ContainerDied","Data":"fcf63073a5ab748224126256937d580a629df2dccf20a09dd98d64e839c0f453"}
Oct 31 00:18:50.918719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:50.918374    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcf63073a5ab748224126256937d580a629df2dccf20a09dd98d64e839c0f453"
Oct 31 00:18:51.348609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.348537    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"]
Oct 31 00:18:51.349364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349348    5078 cpu_manager.go:414] "containerMap: removing container" podUID="497275d4-4d81-4d69-8c6b-b3daec08ffc0" containerName="revoke-keys"
Oct 31 00:18:51.349428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349367    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497275d4-4d81-4d69-8c6b-b3daec08ffc0" containerName="revoke-keys"
Oct 31 00:18:51.349428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349380    5078 cpu_manager.go:414] "containerMap: removing container" podUID="19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" containerName="revoke-keys"
Oct 31 00:18:51.349428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349385    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" containerName="revoke-keys"
Oct 31 00:18:51.349428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349399    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5722e205-4aab-4446-b1ea-a244b5f3da75" containerName="revoke-keys"
Oct 31 00:18:51.349428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349404    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5722e205-4aab-4446-b1ea-a244b5f3da75" containerName="revoke-keys"
Oct 31 00:18:51.349626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349561    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="497275d4-4d81-4d69-8c6b-b3daec08ffc0" containerName="revoke-keys"
Oct 31 00:18:51.349626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349576    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" containerName="revoke-keys"
Oct 31 00:18:51.349626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.349586    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5722e205-4aab-4446-b1ea-a244b5f3da75" containerName="revoke-keys"
Oct 31 00:18:51.354368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.354332    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.380487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.379050    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"]
Oct 31 00:18:51.402485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.390875    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:18:51.451835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.451802    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72sv\" (UniqueName: \"kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.452070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.452052    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.452259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.452234    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.481449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.481423    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:51.554148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.554116    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72sv\" (UniqueName: \"kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.554360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.554347    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.554515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.554499    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.555089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.555070    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.557072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.557044    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.569038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.569017    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72sv\" (UniqueName: \"kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv\") pod \"maas-api-revoke-sub-1b1d8259-8ptng\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:51.714075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:51.714030    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:52.021414 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:52.021383    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f711ec_24e6_4e9d_9d42_abb2d9b65f92.slice/crio-f00968d6b0eacbda9903564143d5688e8bc1ca622537f9743e50fda744b6f7e0 WatchSource:0}: Error finding container f00968d6b0eacbda9903564143d5688e8bc1ca622537f9743e50fda744b6f7e0: Status 404 returned error can't find the container with id f00968d6b0eacbda9903564143d5688e8bc1ca622537f9743e50fda744b6f7e0
Oct 31 00:18:52.024696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:52.024670    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"]
Oct 31 00:18:52.400506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:52.399183    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:18:52.417514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:52.417453    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:18:52.938539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:52.938504    5078 generic.go:356] "Generic (PLEG): container finished" podID="61f711ec-24e6-4e9d-9d42-abb2d9b65f92" containerID="9b1dc70e0def60076240d1055cfda405d06017608902f9b8031780102addfbe7" exitCode=0
Oct 31 00:18:52.938677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:52.938629    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng" event={"ID":"61f711ec-24e6-4e9d-9d42-abb2d9b65f92","Type":"ContainerDied","Data":"9b1dc70e0def60076240d1055cfda405d06017608902f9b8031780102addfbe7"}
Oct 31 00:18:52.938741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:52.938727    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng" event={"ID":"61f711ec-24e6-4e9d-9d42-abb2d9b65f92","Type":"ContainerStarted","Data":"f00968d6b0eacbda9903564143d5688e8bc1ca622537f9743e50fda744b6f7e0"}
Oct 31 00:18:53.471312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.471280    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11079-21bf41"]
Oct 31 00:18:53.478973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.478947    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:18:53.484516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.484491    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11079-21bf41"]
Oct 31 00:18:53.493254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.493232    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lr8lt\""
Oct 31 00:18:53.596817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.595881    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjpm\" (UniqueName: \"kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm\") pod \"test-curl-11079-21bf41\" (UID: \"eea06ac2-ebd2-4c4a-bc49-5a7904766460\") " pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:18:53.697343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.697310    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjpm\" (UniqueName: \"kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm\") pod \"test-curl-11079-21bf41\" (UID: \"eea06ac2-ebd2-4c4a-bc49-5a7904766460\") " pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:18:53.711004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.710974    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjpm\" (UniqueName: \"kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm\") pod \"test-curl-11079-21bf41\" (UID: \"eea06ac2-ebd2-4c4a-bc49-5a7904766460\") " pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:18:53.798690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:53.798590    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:18:54.109298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.096936    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11079-21bf41"]
Oct 31 00:18:54.184883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.184855    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:54.318091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.318045    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca\") pod \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") "
Oct 31 00:18:54.318091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.318094    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca\") pod \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") "
Oct 31 00:18:54.318295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.318123    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv\" (UniqueName: \"kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv\") pod \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\" (UID: \"61f711ec-24e6-4e9d-9d42-abb2d9b65f92\") "
Oct 31 00:18:54.318404 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.318375    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca" pod "61f711ec-24e6-4e9d-9d42-abb2d9b65f92" (UID: "61f711ec-24e6-4e9d-9d42-abb2d9b65f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:54.319192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.319161    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:54.320635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.320614    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca" pod "61f711ec-24e6-4e9d-9d42-abb2d9b65f92" (UID: "61f711ec-24e6-4e9d-9d42-abb2d9b65f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:54.320722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.320701    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv" pod "61f711ec-24e6-4e9d-9d42-abb2d9b65f92" (UID: "61f711ec-24e6-4e9d-9d42-abb2d9b65f92"). InnerVolumeSpecName "kube-api-access-x72sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:54.420743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.420708    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:54.420743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.420738    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72sv\" (UniqueName: \"kubernetes.io/projected/61f711ec-24e6-4e9d-9d42-abb2d9b65f92-kube-api-access-x72sv\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:54.967727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.967694    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng" event={"ID":"61f711ec-24e6-4e9d-9d42-abb2d9b65f92","Type":"ContainerDied","Data":"f00968d6b0eacbda9903564143d5688e8bc1ca622537f9743e50fda744b6f7e0"}
Oct 31 00:18:54.968206 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.967751    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f00968d6b0eacbda9903564143d5688e8bc1ca622537f9743e50fda744b6f7e0"
Oct 31 00:18:54.968206 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.967839    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"
Oct 31 00:18:54.970312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:54.970275    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11079-21bf41" event={"ID":"eea06ac2-ebd2-4c4a-bc49-5a7904766460","Type":"ContainerStarted","Data":"7632e4316f0195c3f4c3f586f21eddcf224454784fef69d6f7e6212a747798f2"}
Oct 31 00:18:57.141151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.141120    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"]
Oct 31 00:18:57.142966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.142924    5078 cpu_manager.go:414] "containerMap: removing container" podUID="61f711ec-24e6-4e9d-9d42-abb2d9b65f92" containerName="revoke-keys"
Oct 31 00:18:57.142966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.142955    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f711ec-24e6-4e9d-9d42-abb2d9b65f92" containerName="revoke-keys"
Oct 31 00:18:57.143163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.143153    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f711ec-24e6-4e9d-9d42-abb2d9b65f92" containerName="revoke-keys"
Oct 31 00:18:57.159605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.159576    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"]
Oct 31 00:18:57.159776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.159698    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.167119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.166945    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2d8f0cae-openshift-default-dockercfg-wt5wq\""
Oct 31 00:18:57.372489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.287808    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.372489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.287952    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjkn\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.411679    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.411871    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.412017    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.412058    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.412234    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.412322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.412947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.412398    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515150    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515209    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515325    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515391    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515444    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515573    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515660    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjkn\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515733    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.516251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.515824    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.532641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.532415    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.541130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.534843    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.541130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.535062    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.541364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.535629    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.541364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.536452    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.541364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.540570    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.560725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.560684    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.566298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.566266    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjkn\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.567140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.567093    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:57.783493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:57.783386    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:58.145252 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:58.145219    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fce4ea1_f092_421b_beee_1357420285ca.slice/crio-8781d6d50211c34d93fef6822d8c383d8ba4fe9554159fb17e7e37717f4f64b2 WatchSource:0}: Error finding container 8781d6d50211c34d93fef6822d8c383d8ba4fe9554159fb17e7e37717f4f64b2: Status 404 returned error can't find the container with id 8781d6d50211c34d93fef6822d8c383d8ba4fe9554159fb17e7e37717f4f64b2
Oct 31 00:18:58.149543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:58.149510    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"]
Oct 31 00:18:58.152141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:58.152105    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:18:58.152259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:58.152177    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:18:58.152259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:58.152210    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:18:59.025414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.025359    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" event={"ID":"1fce4ea1-f092-421b-beee-1357420285ca","Type":"ContainerStarted","Data":"fb335a620d69b33e2ce591c448630fd64fa74fdd5f7c3f521aa6ac1f2565ae04"}
Oct 31 00:18:59.025667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.025433    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" event={"ID":"1fce4ea1-f092-421b-beee-1357420285ca","Type":"ContainerStarted","Data":"8781d6d50211c34d93fef6822d8c383d8ba4fe9554159fb17e7e37717f4f64b2"}
Oct 31 00:18:59.027651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.027624    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11079-21bf41" event={"ID":"eea06ac2-ebd2-4c4a-bc49-5a7904766460","Type":"ContainerStarted","Data":"9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89"}
Oct 31 00:18:59.068102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.068047    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" podStartSLOduration=2.06803391 podStartE2EDuration="2.06803391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:59.062622744 +0000 UTC m=+1507.008993463" watchObservedRunningTime="2026-10-31 00:18:59.06803391 +0000 UTC m=+1507.014404627"
Oct 31 00:18:59.090904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.090858    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11079-21bf41" podStartSLOduration=1.620079697 podStartE2EDuration="6.09084438s" totalImagesPullingTime="4.470764683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:59.089717675 +0000 UTC m=+1507.036088394" watchObservedRunningTime="2026-10-31 00:18:59.09084438 +0000 UTC m=+1507.037215091"
Oct 31 00:18:59.259855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.259799    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"]
Oct 31 00:18:59.267835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.267803    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.271526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.270247    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"]
Oct 31 00:18:59.273357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.273336    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:59.280587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.274144    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:18:59.412213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.412173    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.412750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.412723    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cg8\" (UniqueName: \"kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.413916 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.413882    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.515982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.515951    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cg8\" (UniqueName: \"kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.515982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.516001    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.516253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.516124    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.517018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.516996    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.519057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.519034    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.532191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.532119    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cg8\" (UniqueName: \"kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8\") pod \"maas-api-revoke-sub-9bfeb759-dd88b\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.615675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.615646    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:18:59.786232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.786041    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:59.791577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.791548    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:18:59.927262 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:18:59.927228    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713171b0_70e3_4006_9ab6_86e6d5c72505.slice/crio-e36feea1a5a789e3c4b63ab9031f27fed03319d53f3b6707bfefaad30dad2486 WatchSource:0}: Error finding container e36feea1a5a789e3c4b63ab9031f27fed03319d53f3b6707bfefaad30dad2486: Status 404 returned error can't find the container with id e36feea1a5a789e3c4b63ab9031f27fed03319d53f3b6707bfefaad30dad2486
Oct 31 00:18:59.931453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.931429    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"]
Oct 31 00:18:59.983998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.983947    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11079-21bf41"]
Oct 31 00:18:59.992737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:18:59.992709    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11079-21bf41"]
Oct 31 00:19:00.118768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.117398    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b" event={"ID":"713171b0-70e3-4006-9ab6-86e6d5c72505","Type":"ContainerStarted","Data":"e36feea1a5a789e3c4b63ab9031f27fed03319d53f3b6707bfefaad30dad2486"}
Oct 31 00:19:00.118768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.118328    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:19:00.119311 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.119292    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:19:00.827807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.827725    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"]
Oct 31 00:19:00.829408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.829384    5078 cpu_manager.go:414] "containerMap: removing container" podUID="eea06ac2-ebd2-4c4a-bc49-5a7904766460" containerName="test-curl-11079-21bf41"
Oct 31 00:19:00.829408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.829402    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea06ac2-ebd2-4c4a-bc49-5a7904766460" containerName="test-curl-11079-21bf41"
Oct 31 00:19:00.829606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.829580    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea06ac2-ebd2-4c4a-bc49-5a7904766460" containerName="test-curl-11079-21bf41"
Oct 31 00:19:00.834209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.833776    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.843300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.843273    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"]
Oct 31 00:19:00.844387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.844368    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3f561470-openshift-default-dockercfg-vzvj8\""
Oct 31 00:19:00.929248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929213    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929268    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929328    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nzw\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929395    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929452    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929646    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929757 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929739    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929792    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:00.929853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:00.929842    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.045819    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.045868    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.045920    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.045981    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.046027    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.046371    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.046413    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.046486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.046482    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nzw\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.047076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.046562    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.081534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.064708    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.081534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.064997    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.081534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.065234    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.081534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.065480    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.081534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.066114    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.085025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.084989    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.085259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.085239    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nzw\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.110486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.091348    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.111494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.111408    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token\") pod \"e2e-ait-3f561470-openshift-default-75989964cc-lgsqv\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") " pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.164102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.164072    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:01.246523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.242611    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b" event={"ID":"713171b0-70e3-4006-9ab6-86e6d5c72505","Type":"ContainerStarted","Data":"3d2c9c87781a4bd0b854a0351b4494287221841182eca26ddb33ae04cdbfed1a"}
Oct 31 00:19:01.246523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.242765    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11079-21bf41" podUID="eea06ac2-ebd2-4c4a-bc49-5a7904766460" containerName="test-curl-11079-21bf41" containerID="cri-o://9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89" gracePeriod=2
Oct 31 00:19:01.346009 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.345928    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b" podStartSLOduration=2.345904628 podStartE2EDuration="2.345904628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:01.272315252 +0000 UTC m=+1509.218685969" watchObservedRunningTime="2026-10-31 00:19:01.345904628 +0000 UTC m=+1509.292275346"
Oct 31 00:19:01.672914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.672565    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"]
Oct 31 00:19:01.683972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.683910    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:19:01.684109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.684015    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:19:01.684109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:01.684049    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:19:02.257208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:02.257170    5078 generic.go:356] "Generic (PLEG): container finished" podID="713171b0-70e3-4006-9ab6-86e6d5c72505" containerID="3d2c9c87781a4bd0b854a0351b4494287221841182eca26ddb33ae04cdbfed1a" exitCode=0
Oct 31 00:19:02.257668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:02.257337    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b" event={"ID":"713171b0-70e3-4006-9ab6-86e6d5c72505","Type":"ContainerDied","Data":"3d2c9c87781a4bd0b854a0351b4494287221841182eca26ddb33ae04cdbfed1a"}
Oct 31 00:19:02.259416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:02.259390    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" event={"ID":"c5986930-8d0d-4a08-8790-f49bcd74c771","Type":"ContainerStarted","Data":"707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"}
Oct 31 00:19:02.259592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:02.259446    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" event={"ID":"c5986930-8d0d-4a08-8790-f49bcd74c771","Type":"ContainerStarted","Data":"46cc61893154130a548d823609e5d6d44d5d3eba2204f1f0131c84d250b61f52"}
Oct 31 00:19:02.429489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:02.421572    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" podStartSLOduration=2.42154876 podStartE2EDuration="2.42154876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:02.353496687 +0000 UTC m=+1510.299867403" watchObservedRunningTime="2026-10-31 00:19:02.42154876 +0000 UTC m=+1510.367919478"
Oct 31 00:19:03.165651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.165618    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:03.172366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.172328    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:03.268913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.268884    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:03.270043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.270023    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:19:03.637840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.637810    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:19:03.710125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.710099    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:19:03.803489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.801193    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca\") pod \"713171b0-70e3-4006-9ab6-86e6d5c72505\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") "
Oct 31 00:19:03.803489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.801313    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca\") pod \"713171b0-70e3-4006-9ab6-86e6d5c72505\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") "
Oct 31 00:19:03.803489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.801440    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm\" (UniqueName: \"kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm\") pod \"eea06ac2-ebd2-4c4a-bc49-5a7904766460\" (UID: \"eea06ac2-ebd2-4c4a-bc49-5a7904766460\") "
Oct 31 00:19:03.803489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.802411    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca" pod "713171b0-70e3-4006-9ab6-86e6d5c72505" (UID: "713171b0-70e3-4006-9ab6-86e6d5c72505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:03.803489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.802445    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8\" (UniqueName: \"kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8\") pod \"713171b0-70e3-4006-9ab6-86e6d5c72505\" (UID: \"713171b0-70e3-4006-9ab6-86e6d5c72505\") "
Oct 31 00:19:03.803489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.803487    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:03.805706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.805683    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca" pod "713171b0-70e3-4006-9ab6-86e6d5c72505" (UID: "713171b0-70e3-4006-9ab6-86e6d5c72505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:03.806882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.806612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8" pod "713171b0-70e3-4006-9ab6-86e6d5c72505" (UID: "713171b0-70e3-4006-9ab6-86e6d5c72505"). InnerVolumeSpecName "kube-api-access-t6cg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:03.807054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.806896    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm" pod "eea06ac2-ebd2-4c4a-bc49-5a7904766460" (UID: "eea06ac2-ebd2-4c4a-bc49-5a7904766460"). InnerVolumeSpecName "kube-api-access-phjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:03.905242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.905204    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phjpm\" (UniqueName: \"kubernetes.io/projected/eea06ac2-ebd2-4c4a-bc49-5a7904766460-kube-api-access-phjpm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:03.905242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.905241    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6cg8\" (UniqueName: \"kubernetes.io/projected/713171b0-70e3-4006-9ab6-86e6d5c72505-kube-api-access-t6cg8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:03.905454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:03.905254    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713171b0-70e3-4006-9ab6-86e6d5c72505-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:04.198240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.198190    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea06ac2-ebd2-4c4a-bc49-5a7904766460" path="/var/lib/kubelet/pods/eea06ac2-ebd2-4c4a-bc49-5a7904766460/volumes"
Oct 31 00:19:04.280708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.280673    5078 generic.go:356] "Generic (PLEG): container finished" podID="eea06ac2-ebd2-4c4a-bc49-5a7904766460" containerID="9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89" exitCode=137
Oct 31 00:19:04.281133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.280724    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-21bf41"
Oct 31 00:19:04.281133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.280769    5078 scope.go:122] "RemoveContainer" containerID="9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89"
Oct 31 00:19:04.282605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.282574    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b" event={"ID":"713171b0-70e3-4006-9ab6-86e6d5c72505","Type":"ContainerDied","Data":"e36feea1a5a789e3c4b63ab9031f27fed03319d53f3b6707bfefaad30dad2486"}
Oct 31 00:19:04.283227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.282652    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e36feea1a5a789e3c4b63ab9031f27fed03319d53f3b6707bfefaad30dad2486"
Oct 31 00:19:04.283227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.282588    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"
Oct 31 00:19:04.296248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.296226    5078 scope.go:122] "RemoveContainer" containerID="9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89"
Oct 31 00:19:04.296646 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:04.296624    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89\": container with ID starting with 9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89 not found: ID does not exist" containerID="9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89"
Oct 31 00:19:04.296735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:04.296655    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89"} err="failed to get container status \"9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89\": rpc error: code = NotFound desc = could not find container \"9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89\": container with ID starting with 9db8da8bc6c32d022abefbc8b19f0286cd9213d966925a96fbdb9261bb8a6d89 not found: ID does not exist"
Oct 31 00:19:05.529850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.529820    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"]
Oct 31 00:19:05.530693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.530674    5078 cpu_manager.go:414] "containerMap: removing container" podUID="713171b0-70e3-4006-9ab6-86e6d5c72505" containerName="revoke-keys"
Oct 31 00:19:05.530754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.530694    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713171b0-70e3-4006-9ab6-86e6d5c72505" containerName="revoke-keys"
Oct 31 00:19:05.530838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.530819    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="713171b0-70e3-4006-9ab6-86e6d5c72505" containerName="revoke-keys"
Oct 31 00:19:05.535709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.535690    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.549289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.549268    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3f561470\""
Oct 31 00:19:05.549428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.549407    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3f561470-dockercfg-p7hxb\""
Oct 31 00:19:05.551376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.551357    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"]
Oct 31 00:19:05.636546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.634227    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.636546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.634397    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wdl\" (UniqueName: \"kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl\") pod \"payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.741805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.741762    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wdl\" (UniqueName: \"kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl\") pod \"payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.743778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.743749    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.744878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.744764    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.781578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.781074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wdl\" (UniqueName: \"kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl\") pod \"payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:05.853485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:05.853292    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:06.255340 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:19:06.255291    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7440c139_71fc_4ca0_a5a1_badd10be7aa4.slice/crio-3b92a04af0669a5ca394d120d49232ee339c033474d3600d7609f957e2a0454e WatchSource:0}: Error finding container 3b92a04af0669a5ca394d120d49232ee339c033474d3600d7609f957e2a0454e: Status 404 returned error can't find the container with id 3b92a04af0669a5ca394d120d49232ee339c033474d3600d7609f957e2a0454e
Oct 31 00:19:06.258655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:06.258631    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"]
Oct 31 00:19:06.313242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:06.313165    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn" event={"ID":"7440c139-71fc-4ca0-a5a1-badd10be7aa4","Type":"ContainerStarted","Data":"3b92a04af0669a5ca394d120d49232ee339c033474d3600d7609f957e2a0454e"}
Oct 31 00:19:06.449613 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:06.449581    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:19:07.370886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.370851    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"]
Oct 31 00:19:07.437109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.437074    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"]
Oct 31 00:19:07.437565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.437535    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn" event={"ID":"7440c139-71fc-4ca0-a5a1-badd10be7aa4","Type":"ContainerStarted","Data":"a794ef4a250d6388570d783500329c831c525a6a024abe926dd9d166d4bfcfb5"}
Oct 31 00:19:07.437802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.437782    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:07.437977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.437419    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.444283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.444233    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn" podStartSLOduration=2.444217193 podStartE2EDuration="2.444217193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:07.430249449 +0000 UTC m=+1515.376620168" watchObservedRunningTime="2026-10-31 00:19:07.444217193 +0000 UTC m=+1515.390587910"
Oct 31 00:19:07.457170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.457133    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.491526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.489127    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.491526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.489258    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79h4\" (UniqueName: \"kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.510116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.510082    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:19:07.510116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.510111    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:19:07.600855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.591725    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.600855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.591833    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79h4\" (UniqueName: \"kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.600855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.591895    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.608179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.608132    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.627584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.618490    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.627584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.624290    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79h4\" (UniqueName: \"kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4\") pod \"maas-api-revoke-sub-9e29a7d9-7r7n5\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:07.812119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:07.812089    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:08.103404 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:19:08.103375    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ed62e6_c3fb_44b7_b0ad_2733b5a5c4d7.slice/crio-9eb5e9832457f2b1cdc2a8ff46286739597d872d4632f40d65c70897ae4e6647 WatchSource:0}: Error finding container 9eb5e9832457f2b1cdc2a8ff46286739597d872d4632f40d65c70897ae4e6647: Status 404 returned error can't find the container with id 9eb5e9832457f2b1cdc2a8ff46286739597d872d4632f40d65c70897ae4e6647
Oct 31 00:19:08.106222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:08.106197    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"]
Oct 31 00:19:08.403046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:08.403016    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5" event={"ID":"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7","Type":"ContainerStarted","Data":"9eb5e9832457f2b1cdc2a8ff46286739597d872d4632f40d65c70897ae4e6647"}
Oct 31 00:19:09.414220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:09.414182    5078 generic.go:356] "Generic (PLEG): container finished" podID="68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" containerID="db69a05172307e96849f2753575e6c3e38796c2fa53100d65d6f861e4283d39d" exitCode=0
Oct 31 00:19:09.414688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:09.414270    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5" event={"ID":"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7","Type":"ContainerDied","Data":"db69a05172307e96849f2753575e6c3e38796c2fa53100d65d6f861e4283d39d"}
Oct 31 00:19:10.807627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:10.807578    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.034598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.034550    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.131483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.129520    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:11.131483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.130100    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.233488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.230618    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.469496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.464111    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.485171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.485128    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.522512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.489545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.523072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.521702    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:11.546490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.539889    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.628489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.624636    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:11.842547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.835778    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:19:11.842547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.836735    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:19:11.917486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:11.915423    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:12.173633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.124435    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:12.173633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.125937    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:19:12.173633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.142803    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:12.256219 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.256168    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:12.348492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.323210    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:19:12.349676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.349650    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:19:12.567077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.566966    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:12.645700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.645641    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:12.687310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.590573    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:19:12.744180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.744129    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:19:12.778082 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.764908    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:12.778994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.778955    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:12.917689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.917619    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:12.993478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.993417    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:12.993719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.987117    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:12.993887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.993865    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:12.993993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:12.948251    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.015552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.001225    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:13.075506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.017918    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 31 00:19:13.075506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.058198    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.105543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.076093    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:13.138712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.138664    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.139425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.139390    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:13.229545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.209948    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.267857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.259081    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.398490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.385812    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:13.398490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.385874    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.766535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.765713    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.821931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.821887    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:13.894902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.841013    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:13.987290    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.026887    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.027351    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.063387    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.082554    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.085565    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.129140    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.175552    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:14.217546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.175667    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:14.317490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.283010    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.362002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.361955    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.362186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.292659    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.383491 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:14.363826    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.014s"
Oct 31 00:19:14.525655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.490131    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:19:14.525655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.490241    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:19:14.645490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.616881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.698489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.690762    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:14.698489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.697839    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.816552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.816178    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:14.816552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.816252    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.882489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.870609    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:14.906668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.906608    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 31 00:19:14.979684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:14.979630    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:15.005762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.005711    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.030779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.001164    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.031701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.031653    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:15.073745 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.073630    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.139502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.131369    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:19:15.235490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.210752    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:19:15.288660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.236195    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:15.288934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.288901    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.289080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.237566    5078 trace.go:236] Trace[1628869970]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" (31-Oct-2026 00:19:14.170) (total time: 1067ms):
Oct 31 00:19:15.289080 ip-10-0-143-16 kubenswrapper[5078]: Trace[1628869970]: [1.067342427s] [1.067342427s] END
Oct 31 00:19:15.365178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.339559    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:19:15.399548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.399505    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.642490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.405149    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.667571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.405455    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:15.711597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.405509    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:15.722613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.475117    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.723080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.476153    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:19:15.723428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.476189    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 31 00:19:15.735756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.482748    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:19:15.736000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.735983    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:19:15.736190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.736175    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:19:15.736652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.525966    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.762491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.533806    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Oct 31 00:19:15.763477 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.549958    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:15.763708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.763672    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.763833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.534603    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:15.764201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.534061    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:19:15.764576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.577802    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:15.764747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.764725    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.764887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.763025    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:15.765026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.765007    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:15.765177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.739114    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:15.873683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.873650    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:15.956565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.954656    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 31 00:19:15.958961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.958932    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"]
Oct 31 00:19:15.960356 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:15.960331    5078 cpu_manager.go:414] "containerMap: removing container" podUID="68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" containerName="revoke-keys"
Oct 31 00:19:16.235634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.235542    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" containerName="revoke-keys"
Oct 31 00:19:16.263513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.236525    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" containerName="revoke-keys"
Oct 31 00:19:16.282517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.169630    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:16.336919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.336879    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4\" (UniqueName: \"kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4\") pod \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") "
Oct 31 00:19:16.357905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.337484    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca\") pod \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") "
Oct 31 00:19:16.365738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.365707    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca\") pod \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\" (UID: \"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7\") "
Oct 31 00:19:16.409883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.409845    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:16.411299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.411264    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:16.450096 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.340116    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.485528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.459091    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:16.486633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.340002    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5" event={"ID":"68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7","Type":"ContainerDied","Data":"9eb5e9832457f2b1cdc2a8ff46286739597d872d4632f40d65c70897ae4e6647"}
Oct 31 00:19:16.637648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.637608    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb5e9832457f2b1cdc2a8ff46286739597d872d4632f40d65c70897ae4e6647"
Oct 31 00:19:16.637913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.637893    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"]
Oct 31 00:19:16.638032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.544334    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 00:19:16.714519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.575267    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.714519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.689089    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.714519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.689141    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.714519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.689201    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86pm\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.714519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.689255    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.714519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.689368    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.715577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.715526    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:16.745472 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.544277    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 00:19:16.764930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.658535    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:16.955735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.880637    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:16.955735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.885743    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:16.960567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.960522    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:19:17.106692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.106587    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:17.108143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.108110    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:19:17.261668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.261636    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w2ptn\""
Oct 31 00:19:17.408839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.487050    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca" pod "68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" (UID: "68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:17.409059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.544931    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca" pod "68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" (UID: "68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:17.409170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:16.555374    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:17.409322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.409294    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:17.409488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.301938    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 00:19:17.413522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.271556    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:17.427402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.271620    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:17.456602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.271678    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:19:17.521271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.515679    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:17.542642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.378269    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4" pod "68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" (UID: "68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7"). InnerVolumeSpecName "kube-api-access-m79h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:17.543256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.317510    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.543576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.543541    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86pm\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.543843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.543823    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.544109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.544092    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.567849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.567799    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:19:17.732146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.318399    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.732427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.651559    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.792378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.792342    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:17.894495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.874727    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.026573 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:17.920746    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.110284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.110237    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.354546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.320606    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86pm\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.410598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.110869    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:18.467500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.225571    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m79h4\" (UniqueName: \"kubernetes.io/projected/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-kube-api-access-m79h4\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.467500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.466665    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.467500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.466690    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.492374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.298736    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.515485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.070799    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.363242    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.389846    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.110664    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.433067    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.434763    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.450665    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.456715    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.298681    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:18.577750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.354960    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.620536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.611944    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.631918 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.621178    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.695512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.648971    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qnjc2\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:18.759527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.751659    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:18.837489 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:18.827878    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.149s"
Oct 31 00:19:18.838162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.761068    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:18.859619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.799227    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"
Oct 31 00:19:18.901364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:18.901330    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:19.194696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.194598    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:19.403644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.403596    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:19.495557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.491830    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:19.496504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.496439    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:19.564129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.564068    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:19:19.564293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.564131    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:19.635647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.630711    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.711507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.711441    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded"
Oct 31 00:19:19.755330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.747897    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.801909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.801882    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:19:19.844575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.844534    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:19.851436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.851402    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.852231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.852204    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.857613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.857595    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:19:19.858014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.857925    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.858820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.858780    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.863693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.863663    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:19:19.880313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.880293    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:19:19.931486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.931436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:19:19.955378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.955347    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:19:19.955696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.955682    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:19:19.955827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.955817    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:19:19.966969 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:19.966024    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ed62e6_c3fb_44b7_b0ad_2733b5a5c4d7.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:19:19.968284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.968230    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:19:19.968284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.968282    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:19:19.968489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.968318    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:19:19.995401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.995318    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:19:19.995683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:19.995483    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:19:20.003095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:20.003073    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:19:20.244991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:20.244919    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"]
Oct 31 00:19:20.959296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:20.959240    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" event={"ID":"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8","Type":"ContainerStarted","Data":"fe0c4db43c69c080f77e754ff025c5e77a9d6df5593b59f42ca263c083f7b390"}
Oct 31 00:19:24.331916 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:24.331882    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:19:24.332204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:24.331948    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:19:25.108523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:25.108482    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" event={"ID":"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8","Type":"ContainerStarted","Data":"ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8"}
Oct 31 00:19:25.110113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:25.109848    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:25.111359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:25.111337    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qnjc2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:19:25.111962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:25.111925    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:19:25.157162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:25.157111    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" podStartSLOduration=9.078999811 podStartE2EDuration="13.15709853s" totalImagesPullingTime="4.078098719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:25.15208162 +0000 UTC m=+1533.098452340" watchObservedRunningTime="2026-10-31 00:19:25.15709853 +0000 UTC m=+1533.103469246"
Oct 31 00:19:26.119712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:26.119685    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:19:36.390738 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:36.390690    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:19:37.051611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.051575    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"]
Oct 31 00:19:37.060635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.060607    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.064289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.064261    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"]
Oct 31 00:19:37.071747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.071725    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:19:37.073355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.072014    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:19:37.214233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.214200    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.214408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.214254    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.214408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.214357    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8kc\" (UniqueName: \"kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.315909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.315810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.316084 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.315978    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.316358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.316320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8kc\" (UniqueName: \"kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.316654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.316626    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.319149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.319130    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.329508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.329484    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8kc\" (UniqueName: \"kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc\") pod \"maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.393045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.393012    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:37.700356 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:19:37.700322    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8b70e1_d446_48f2_9360_4971b4bd7a90.slice/crio-07340f628ecd3fd0efe58fe46563cf7c9f0f43735cef7c8c5c344a8b7f536ac4 WatchSource:0}: Error finding container 07340f628ecd3fd0efe58fe46563cf7c9f0f43735cef7c8c5c344a8b7f536ac4: Status 404 returned error can't find the container with id 07340f628ecd3fd0efe58fe46563cf7c9f0f43735cef7c8c5c344a8b7f536ac4
Oct 31 00:19:37.705692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:37.705668    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"]
Oct 31 00:19:38.445346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:38.445283    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4" event={"ID":"3c8b70e1-d446-48f2-9360-4971b4bd7a90","Type":"ContainerStarted","Data":"0d50cc426fd6b1287ea7d8e44bcda5a7d4a2154b60ed71e0b56c4000a4c551de"}
Oct 31 00:19:38.445850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:38.445375    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4" event={"ID":"3c8b70e1-d446-48f2-9360-4971b4bd7a90","Type":"ContainerStarted","Data":"07340f628ecd3fd0efe58fe46563cf7c9f0f43735cef7c8c5c344a8b7f536ac4"}
Oct 31 00:19:38.474053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:38.473996    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4" podStartSLOduration=1.473984421 podStartE2EDuration="1.473984421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:38.466132443 +0000 UTC m=+1546.412503163" watchObservedRunningTime="2026-10-31 00:19:38.473984421 +0000 UTC m=+1546.420355182"
Oct 31 00:19:39.457743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:39.457713    5078 generic.go:356] "Generic (PLEG): container finished" podID="3c8b70e1-d446-48f2-9360-4971b4bd7a90" containerID="0d50cc426fd6b1287ea7d8e44bcda5a7d4a2154b60ed71e0b56c4000a4c551de" exitCode=0
Oct 31 00:19:39.458189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:39.457811    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4" event={"ID":"3c8b70e1-d446-48f2-9360-4971b4bd7a90","Type":"ContainerDied","Data":"0d50cc426fd6b1287ea7d8e44bcda5a7d4a2154b60ed71e0b56c4000a4c551de"}
Oct 31 00:19:40.622309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.607768    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"]
Oct 31 00:19:40.631062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.631034    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.638618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.637943    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"]
Oct 31 00:19:40.746432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.746408    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:40.796508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.796426    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.796690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.796593    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddnw\" (UniqueName: \"kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.796690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.796669    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.900931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.900897    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc\" (UniqueName: \"kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc\") pod \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") "
Oct 31 00:19:40.901102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.900981    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca\") pod \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") "
Oct 31 00:19:40.901102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.901028    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca\") pod \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\" (UID: \"3c8b70e1-d446-48f2-9360-4971b4bd7a90\") "
Oct 31 00:19:40.901421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.901385    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca" pod "3c8b70e1-d446-48f2-9360-4971b4bd7a90" (UID: "3c8b70e1-d446-48f2-9360-4971b4bd7a90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:40.901570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.901428    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.902331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.902302    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddnw\" (UniqueName: \"kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.902829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.902806    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.903878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.903755    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.903989 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.903951    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc" pod "3c8b70e1-d446-48f2-9360-4971b4bd7a90" (UID: "3c8b70e1-d446-48f2-9360-4971b4bd7a90"). InnerVolumeSpecName "kube-api-access-vf8kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:40.904093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.904069    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca" pod "3c8b70e1-d446-48f2-9360-4971b4bd7a90" (UID: "3c8b70e1-d446-48f2-9360-4971b4bd7a90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:40.904153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.904129    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8kc\" (UniqueName: \"kubernetes.io/projected/3c8b70e1-d446-48f2-9360-4971b4bd7a90-kube-api-access-vf8kc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:40.904153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.904141    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:40.904402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.904387    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.921283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.921255    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddnw\" (UniqueName: \"kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw\") pod \"maas-api-revoke-keys-e2e-ait-3f561470-xlzp8\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:40.960866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:40.960838    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:41.006804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:41.006723    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b70e1-d446-48f2-9360-4971b4bd7a90-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:41.257084 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:19:41.257051    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba49e24e_9107_4208_a62b_713ee4b98551.slice/crio-a0a9a502bf75b6756d64c606830d7ec3e7e5fca5c778f3871fd2f8787ec4a293 WatchSource:0}: Error finding container a0a9a502bf75b6756d64c606830d7ec3e7e5fca5c778f3871fd2f8787ec4a293: Status 404 returned error can't find the container with id a0a9a502bf75b6756d64c606830d7ec3e7e5fca5c778f3871fd2f8787ec4a293
Oct 31 00:19:41.260642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:41.260450    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"]
Oct 31 00:19:41.576541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:41.575810    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4" event={"ID":"3c8b70e1-d446-48f2-9360-4971b4bd7a90","Type":"ContainerDied","Data":"07340f628ecd3fd0efe58fe46563cf7c9f0f43735cef7c8c5c344a8b7f536ac4"}
Oct 31 00:19:41.576541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:41.575896    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07340f628ecd3fd0efe58fe46563cf7c9f0f43735cef7c8c5c344a8b7f536ac4"
Oct 31 00:19:41.576541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:41.576025    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"
Oct 31 00:19:41.586534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:41.582185    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8" event={"ID":"ba49e24e-9107-4208-a62b-713ee4b98551","Type":"ContainerStarted","Data":"a0a9a502bf75b6756d64c606830d7ec3e7e5fca5c778f3871fd2f8787ec4a293"}
Oct 31 00:19:42.597518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:42.597097    5078 generic.go:356] "Generic (PLEG): container finished" podID="ba49e24e-9107-4208-a62b-713ee4b98551" containerID="af06753612df4bf0f25ab7b83acea607389d8b8385e011815dfcc61b9004e05b" exitCode=0
Oct 31 00:19:42.597518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:42.597218    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8" event={"ID":"ba49e24e-9107-4208-a62b-713ee4b98551","Type":"ContainerDied","Data":"af06753612df4bf0f25ab7b83acea607389d8b8385e011815dfcc61b9004e05b"}
Oct 31 00:19:43.945333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:43.944915    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:44.066253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.058769    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca\") pod \"ba49e24e-9107-4208-a62b-713ee4b98551\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") "
Oct 31 00:19:44.066253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.058829    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca\") pod \"ba49e24e-9107-4208-a62b-713ee4b98551\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") "
Oct 31 00:19:44.066253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.059120    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw\" (UniqueName: \"kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw\") pod \"ba49e24e-9107-4208-a62b-713ee4b98551\" (UID: \"ba49e24e-9107-4208-a62b-713ee4b98551\") "
Oct 31 00:19:44.090486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.078526    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca" pod "ba49e24e-9107-4208-a62b-713ee4b98551" (UID: "ba49e24e-9107-4208-a62b-713ee4b98551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:44.090486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.086603    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca" pod "ba49e24e-9107-4208-a62b-713ee4b98551" (UID: "ba49e24e-9107-4208-a62b-713ee4b98551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:44.090486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.088019    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw" pod "ba49e24e-9107-4208-a62b-713ee4b98551" (UID: "ba49e24e-9107-4208-a62b-713ee4b98551"). InnerVolumeSpecName "kube-api-access-vddnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.175530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.163628    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.175530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.163663    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba49e24e-9107-4208-a62b-713ee4b98551-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.175530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.163678    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddnw\" (UniqueName: \"kubernetes.io/projected/ba49e24e-9107-4208-a62b-713ee4b98551-kube-api-access-vddnw\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.868487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.851353    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8" event={"ID":"ba49e24e-9107-4208-a62b-713ee4b98551","Type":"ContainerDied","Data":"a0a9a502bf75b6756d64c606830d7ec3e7e5fca5c778f3871fd2f8787ec4a293"}
Oct 31 00:19:44.868487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.851450    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a9a502bf75b6756d64c606830d7ec3e7e5fca5c778f3871fd2f8787ec4a293"
Oct 31 00:19:44.868487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:44.851576    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"
Oct 31 00:19:45.478620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:45.478571    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:19:46.491657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.486693    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:46.491657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.487351    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:19:46.731495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.699602    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:46.792550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.790564    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:46.819811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.696574    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:46.879608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.877736    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:46.879608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.878169    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:46.894529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.889653    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:46.894529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:46.890018    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:19:47.134801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.085568    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.139701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.139661    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.167110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.167070    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.447589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.418938    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Oct 31 00:19:47.580389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.580346    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:19:47.761737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.761615    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.780760    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.782137    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.803703    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.804185    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.839734    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.841135    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:19:47.853631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.841564    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.892614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.892566    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.894186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.894156    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:47.895586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.895560    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:19:47.895724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.895693    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:47.895860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.895846    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:19:47.896010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.895872    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:19:47.932516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.906802    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:19:47.932516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:47.906871    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:19:48.192621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.138679    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:48.442593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.144529    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:48.573547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.239565    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:48.624489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.622540    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:19:48.624489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.622564    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:48.624489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.622603    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:19:48.624489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.622637    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:19:48.707436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.707395    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:48.945714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.945656    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:49.724539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.956548    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:49.727135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.991862    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.730170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:48.992078    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.767708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.015799    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:49.767708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.066760    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"]
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.137333    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.137588    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.137789    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.606881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.630032    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.661630    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.673232    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:49.772452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.698868    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded"
Oct 31 00:19:49.881568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.836197    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ba49e24e-9107-4208-a62b-713ee4b98551" containerName="revoke-keys"
Oct 31 00:19:49.881568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.836229    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49e24e-9107-4208-a62b-713ee4b98551" containerName="revoke-keys"
Oct 31 00:19:49.881568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.836282    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3c8b70e1-d446-48f2-9360-4971b4bd7a90" containerName="revoke-keys"
Oct 31 00:19:49.881568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.836290    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8b70e1-d446-48f2-9360-4971b4bd7a90" containerName="revoke-keys"
Oct 31 00:19:49.881568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.836713    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba49e24e-9107-4208-a62b-713ee4b98551" containerName="revoke-keys"
Oct 31 00:19:49.881568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.836741    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8b70e1-d446-48f2-9360-4971b4bd7a90" containerName="revoke-keys"
Oct 31 00:19:49.958402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.934755    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:19:49.958402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.936195    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:49.991128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:49.991041    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:19:50.165485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.147963    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:19:50.197270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.197226    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.199016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.197929    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.202122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.199196    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrcx\" (UniqueName: \"kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.312072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.311986    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:19:50.334264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.334225    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.334643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.334617    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrcx\" (UniqueName: \"kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.334980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.334957    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.398834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.393605    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:50.398834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.395442    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:19:50.510187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.507305    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:50.712510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.711989    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:50.738221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.734557    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:50.738221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.734748    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:50.738221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.734866    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:19:50.764896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.764644    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:50.785511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.776760    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:50.785511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.776941    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:50.988685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:50.853359    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:51.153368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.004619    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.166526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.004661    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.174487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.004682    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.214621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.004700    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.258538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.004717    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.258538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.075380    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:51.311790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.303313    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrcx\" (UniqueName: \"kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx\") pod \"maas-api-revoke-keys-e2e-ait-2854448d-6dm8d\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:51.514540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.513513    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"]
Oct 31 00:19:51.532497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.515587    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:51.765542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.759096    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:51.788615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.771119    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:51.788615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.771397    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.788615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.771652    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:51.788615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.774959    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:51.815887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.797789    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:51.815887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:51.798258    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.287611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.287563    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.315730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.288164    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.315730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.293559    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.336489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.332589    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.399039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.353364    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.806583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.793834    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.929495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.799571    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:52.937329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.935773    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.937738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.935811    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.951653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.951600    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 31 00:19:52.952053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.952024    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.952345 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.952321    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:52.952651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:52.952627    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:53.310504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:53.304219    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:19:53.326535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:53.324195    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:53.326535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:53.324423    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:53.413504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:53.399344    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:53.451991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:53.451879    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:53.760514 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:19:53.760409    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.512s"
Oct 31 00:19:54.662656 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:19:54.662151    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c44c07_825b_43bf_a795_f1f32dd5019b.slice/crio-6ebcf3f9259e2fa3a93606f8f78bc12b007625348a244de51c14267ddd2e54d2 WatchSource:0}: Error finding container 6ebcf3f9259e2fa3a93606f8f78bc12b007625348a244de51c14267ddd2e54d2: Status 404 returned error can't find the container with id 6ebcf3f9259e2fa3a93606f8f78bc12b007625348a244de51c14267ddd2e54d2
Oct 31 00:19:54.816418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.816369    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:54.817552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.817521    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:54.818388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.818366    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:19:54.832106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.832080    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 31 00:19:54.833878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.833853    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:54.834669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.834646    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:54.838550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.834824    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:19:54.874073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.874034    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:19:54.874332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.874318    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:19:54.874490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.874478    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:19:54.874654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.874644    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:19:54.880599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.880551    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:19:54.895601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.895567    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:54.964628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:54.964531    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:55.291539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.228965    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:19:55.306634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.292896    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 31 00:19:55.817715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.817616    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:55.822487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.818712    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:19:55.841889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.831687    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:55.875485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.837581    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:19:55.875485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.846279    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:55.899612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.899572    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:19:55.952815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.952449    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:55.967280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:55.967239    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:19:56.034005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.031748    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:19:56.318714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.318684    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:19:56.329807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.329771    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"]
Oct 31 00:19:56.330121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.330108    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:19:56.330377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.330358    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"]
Oct 31 00:19:56.330563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.330550    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:19:56.330747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.330730    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:19:56.330877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.330860    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:19:56.331006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.330997    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:19:56.387031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:56.378483    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d" event={"ID":"15c44c07-825b-43bf-a795-f1f32dd5019b","Type":"ContainerStarted","Data":"6ebcf3f9259e2fa3a93606f8f78bc12b007625348a244de51c14267ddd2e54d2"}
Oct 31 00:19:57.419107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:57.419064    5078 generic.go:356] "Generic (PLEG): container finished" podID="15c44c07-825b-43bf-a795-f1f32dd5019b" containerID="50530c2eb575c290e8dc190065789d29fbf2a29e7765a0efc0ee943fd9e036a0" exitCode=0
Oct 31 00:19:57.419603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:57.419196    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d" event={"ID":"15c44c07-825b-43bf-a795-f1f32dd5019b","Type":"ContainerDied","Data":"50530c2eb575c290e8dc190065789d29fbf2a29e7765a0efc0ee943fd9e036a0"}
Oct 31 00:19:58.716527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.713568    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:19:58.744128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.743258    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca\") pod \"15c44c07-825b-43bf-a795-f1f32dd5019b\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") "
Oct 31 00:19:58.744128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.743317    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca\") pod \"15c44c07-825b-43bf-a795-f1f32dd5019b\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") "
Oct 31 00:19:58.744128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.743730    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx\" (UniqueName: \"kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx\") pod \"15c44c07-825b-43bf-a795-f1f32dd5019b\" (UID: \"15c44c07-825b-43bf-a795-f1f32dd5019b\") "
Oct 31 00:19:58.752179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.751204    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx" pod "15c44c07-825b-43bf-a795-f1f32dd5019b" (UID: "15c44c07-825b-43bf-a795-f1f32dd5019b"). InnerVolumeSpecName "kube-api-access-gjrcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:58.752179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.752090    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca" pod "15c44c07-825b-43bf-a795-f1f32dd5019b" (UID: "15c44c07-825b-43bf-a795-f1f32dd5019b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:58.755143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.755105    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca" pod "15c44c07-825b-43bf-a795-f1f32dd5019b" (UID: "15c44c07-825b-43bf-a795-f1f32dd5019b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:58.846098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.846066    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrcx\" (UniqueName: \"kubernetes.io/projected/15c44c07-825b-43bf-a795-f1f32dd5019b-kube-api-access-gjrcx\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:58.846098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.846094    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:58.846098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:58.846105    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15c44c07-825b-43bf-a795-f1f32dd5019b-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:59.460597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:59.460562    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d" event={"ID":"15c44c07-825b-43bf-a795-f1f32dd5019b","Type":"ContainerDied","Data":"6ebcf3f9259e2fa3a93606f8f78bc12b007625348a244de51c14267ddd2e54d2"}
Oct 31 00:19:59.460928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:59.460906    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ebcf3f9259e2fa3a93606f8f78bc12b007625348a244de51c14267ddd2e54d2"
Oct 31 00:19:59.460928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:19:59.460707    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"
Oct 31 00:20:01.159673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:01.159627    5078 patch_prober.go:29] interesting pod/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:20:01.160207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:01.159706    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" podUID="1fce4ea1-f092-421b-beee-1357420285ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:20:01.334986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:01.334895    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn" podUID="7440c139-71fc-4ca0-a5a1-badd10be7aa4" containerName="payload-pre-processing" containerID="cri-o://a794ef4a250d6388570d783500329c831c525a6a024abe926dd9d166d4bfcfb5" gracePeriod=25
Oct 31 00:20:01.500653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:01.496762    5078 generic.go:356] "Generic (PLEG): container finished" podID="7440c139-71fc-4ca0-a5a1-badd10be7aa4" containerID="a794ef4a250d6388570d783500329c831c525a6a024abe926dd9d166d4bfcfb5" exitCode=0
Oct 31 00:20:01.500653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:01.497073    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn" event={"ID":"7440c139-71fc-4ca0-a5a1-badd10be7aa4","Type":"ContainerDied","Data":"a794ef4a250d6388570d783500329c831c525a6a024abe926dd9d166d4bfcfb5"}
Oct 31 00:20:01.892487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:01.886150    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:20:02.036936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.036833    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume\") pod \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") "
Oct 31 00:20:02.036936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.036922    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl\" (UniqueName: \"kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl\") pod \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\" (UID: \"7440c139-71fc-4ca0-a5a1-badd10be7aa4\") "
Oct 31 00:20:02.037375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.037343    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume" pod "7440c139-71fc-4ca0-a5a1-badd10be7aa4" (UID: "7440c139-71fc-4ca0-a5a1-badd10be7aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:02.037560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.037533    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7440c139-71fc-4ca0-a5a1-badd10be7aa4-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:02.040005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.039980    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl" pod "7440c139-71fc-4ca0-a5a1-badd10be7aa4" (UID: "7440c139-71fc-4ca0-a5a1-badd10be7aa4"). InnerVolumeSpecName "kube-api-access-p7wdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:02.139773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.139737    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wdl\" (UniqueName: \"kubernetes.io/projected/7440c139-71fc-4ca0-a5a1-badd10be7aa4-kube-api-access-p7wdl\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:02.514724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.514692    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn" event={"ID":"7440c139-71fc-4ca0-a5a1-badd10be7aa4","Type":"ContainerDied","Data":"3b92a04af0669a5ca394d120d49232ee339c033474d3600d7609f957e2a0454e"}
Oct 31 00:20:02.515586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.515517    5078 scope.go:122] "RemoveContainer" containerID="a794ef4a250d6388570d783500329c831c525a6a024abe926dd9d166d4bfcfb5"
Oct 31 00:20:02.515731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.515715    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"
Oct 31 00:20:02.561846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.561608    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"]
Oct 31 00:20:02.572951 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:02.572916    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f561470-78f586c9d7-9wprn"]
Oct 31 00:20:03.263129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:03.263080    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"]
Oct 31 00:20:03.271118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:03.270167    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2d8f0cae-5hqp4"]
Oct 31 00:20:03.503486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:03.502654    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"]
Oct 31 00:20:03.503486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:03.502908    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" podUID="1fce4ea1-f092-421b-beee-1357420285ca" containerName="istio-proxy" containerID="cri-o://fb335a620d69b33e2ce591c448630fd64fa74fdd5f7c3f521aa6ac1f2565ae04" gracePeriod=30
Oct 31 00:20:04.202954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:04.202918    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8b70e1-d446-48f2-9360-4971b4bd7a90" path="/var/lib/kubelet/pods/3c8b70e1-d446-48f2-9360-4971b4bd7a90/volumes"
Oct 31 00:20:04.203518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:04.203455    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7440c139-71fc-4ca0-a5a1-badd10be7aa4" path="/var/lib/kubelet/pods/7440c139-71fc-4ca0-a5a1-badd10be7aa4/volumes"
Oct 31 00:20:04.309654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:04.309610    5078 patch_prober.go:29] interesting pod/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:20:04.434901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:04.309680    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:20:05.602539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:05.596432    5078 generic.go:356] "Generic (PLEG): container finished" podID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerID="707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29" exitCode=0
Oct 31 00:20:05.602539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:05.596554    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" event={"ID":"c5986930-8d0d-4a08-8790-f49bcd74c771","Type":"ContainerDied","Data":"707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"}
Oct 31 00:20:05.602539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:05.597610    5078 scope.go:122] "RemoveContainer" containerID="707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"
Oct 31 00:20:05.602539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:05.600703    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:20:05.602539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:05.600792    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:20:05.602539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:05.600843    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:20:06.543245 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:20:06.543200    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:20:06.919370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:06.912548    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" event={"ID":"c5986930-8d0d-4a08-8790-f49bcd74c771","Type":"ContainerStarted","Data":"07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53"}
Oct 31 00:20:07.034518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.032894    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"]
Oct 31 00:20:07.043450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.043418    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f561470-xlzp8"]
Oct 31 00:20:07.059276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.059246    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"]
Oct 31 00:20:07.065850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.065817    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00837c1d-28stb"]
Oct 31 00:20:07.164795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.164763    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:20:07.165574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.165551    5078 patch_prober.go:29] interesting pod/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 00:20:07.165724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.165619    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 00:20:07.412495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.412425    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"]
Oct 31 00:20:07.966316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:07.966283    5078 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" secret="" err="secret \"e2e-ait-3f561470-openshift-default-dockercfg-vzvj8\" not found"
Oct 31 00:20:08.165101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:08.165068    5078 patch_prober.go:29] interesting pod/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 00:20:08.165325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:08.165132    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 00:20:08.216422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:08.216326    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9eb8e1-aa8f-4469-a451-d79190c375c3" path="/var/lib/kubelet/pods/7d9eb8e1-aa8f-4469-a451-d79190c375c3/volumes"
Oct 31 00:20:08.217738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:08.217712    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba49e24e-9107-4208-a62b-713ee4b98551" path="/var/lib/kubelet/pods/ba49e24e-9107-4208-a62b-713ee4b98551/volumes"
Oct 31 00:20:09.145393 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:09.143153    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy" containerID="cri-o://07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53" gracePeriod=30
Oct 31 00:20:10.164668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.164634    5078 generic.go:356] "Generic (PLEG): container finished" podID="1fce4ea1-f092-421b-beee-1357420285ca" containerID="fb335a620d69b33e2ce591c448630fd64fa74fdd5f7c3f521aa6ac1f2565ae04" exitCode=0
Oct 31 00:20:10.165291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.164713    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" event={"ID":"1fce4ea1-f092-421b-beee-1357420285ca","Type":"ContainerDied","Data":"fb335a620d69b33e2ce591c448630fd64fa74fdd5f7c3f521aa6ac1f2565ae04"}
Oct 31 00:20:10.223307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.223278    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:20:10.265520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.264394    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.265520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.264546    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.265520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.264622    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.265520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.264652    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.265520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.264753    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.265520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.264783    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.266670    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.266860    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.266928    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data\") pod \"1fce4ea1-f092-421b-beee-1357420285ca\" (UID: \"1fce4ea1-f092-421b-beee-1357420285ca\") "
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.296936    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.297360    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.297665    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.297842    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:10.303383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.298005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:10.343330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.342741    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:10.343970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.343947    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:10.344598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.344570    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:20:10.365408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.365096    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn" pod "1fce4ea1-f092-421b-beee-1357420285ca" (UID: "1fce4ea1-f092-421b-beee-1357420285ca"). InnerVolumeSpecName "kube-api-access-bhjkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:10.424293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424254    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424293    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424309    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424326    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424342    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424358    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4ea1-f092-421b-beee-1357420285ca-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424374    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjkn\" (UniqueName: \"kubernetes.io/projected/1fce4ea1-f092-421b-beee-1357420285ca-kube-api-access-bhjkn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424390    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4ea1-f092-421b-beee-1357420285ca-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:10.424621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:10.424408    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4ea1-f092-421b-beee-1357420285ca-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:11.231779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:11.231750    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"
Oct 31 00:20:11.233803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:11.231773    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4" event={"ID":"1fce4ea1-f092-421b-beee-1357420285ca","Type":"ContainerDied","Data":"8781d6d50211c34d93fef6822d8c383d8ba4fe9554159fb17e7e37717f4f64b2"}
Oct 31 00:20:11.233930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:11.233845    5078 scope.go:122] "RemoveContainer" containerID="fb335a620d69b33e2ce591c448630fd64fa74fdd5f7c3f521aa6ac1f2565ae04"
Oct 31 00:20:11.314179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:11.312675    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"]
Oct 31 00:20:11.319618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:11.319590    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-p47l4"]
Oct 31 00:20:12.213876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:12.213839    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fce4ea1-f092-421b-beee-1357420285ca" path="/var/lib/kubelet/pods/1fce4ea1-f092-421b-beee-1357420285ca/volumes"
Oct 31 00:20:14.556484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:14.556432    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"]
Oct 31 00:20:14.563085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:14.563059    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2854448d-6dm8d"]
Oct 31 00:20:16.198017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:16.197981    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c44c07-825b-43bf-a795-f1f32dd5019b" path="/var/lib/kubelet/pods/15c44c07-825b-43bf-a795-f1f32dd5019b/volumes"
Oct 31 00:20:18.044282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.044259    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"]
Oct 31 00:20:18.052871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.052847    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7fba5-cprd4"]
Oct 31 00:20:18.239002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.238972    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcb0fee-1ff2-4f0f-aef5-882bc22a012e" path="/var/lib/kubelet/pods/bfcb0fee-1ff2-4f0f-aef5-882bc22a012e/volumes"
Oct 31 00:20:18.289539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.281667    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:20:18.310547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.310415    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:20:18.327885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.327857    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.327894    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.327993    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.328019    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.328062    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.328100    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.328169    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.328213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.328212    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.329044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.329016    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:18.329628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.329588    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:18.329832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.329804    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:18.329906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.329857    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:18.330191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.330169    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:18.331789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.331766    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:18.331925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.331781    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:20:18.331925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.331921    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:18.391709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.391672    5078 generic.go:356] "Generic (PLEG): container finished" podID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerID="07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53" exitCode=0
Oct 31 00:20:18.391902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.391799    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"
Oct 31 00:20:18.392003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.391983    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" event={"ID":"c5986930-8d0d-4a08-8790-f49bcd74c771","Type":"ContainerDied","Data":"07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53"}
Oct 31 00:20:18.392071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.392059    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv" event={"ID":"c5986930-8d0d-4a08-8790-f49bcd74c771","Type":"ContainerDied","Data":"46cc61893154130a548d823609e5d6d44d5d3eba2204f1f0131c84d250b61f52"}
Oct 31 00:20:18.392129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.392089    5078 scope.go:122] "RemoveContainer" containerID="07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53"
Oct 31 00:20:18.421701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.421679    5078 scope.go:122] "RemoveContainer" containerID="707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"
Oct 31 00:20:18.432145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.432120    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw\") pod \"c5986930-8d0d-4a08-8790-f49bcd74c771\" (UID: \"c5986930-8d0d-4a08-8790-f49bcd74c771\") "
Oct 31 00:20:18.434919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434893    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434923    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434937    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434950    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434963    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5986930-8d0d-4a08-8790-f49bcd74c771-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434975    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.434988    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.435001    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5986930-8d0d-4a08-8790-f49bcd74c771-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.435482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.435438    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw" pod "c5986930-8d0d-4a08-8790-f49bcd74c771" (UID: "c5986930-8d0d-4a08-8790-f49bcd74c771"). InnerVolumeSpecName "kube-api-access-t5nzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:18.436847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.436811    5078 scope.go:122] "RemoveContainer" containerID="07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53"
Oct 31 00:20:18.437177 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:20:18.437116    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53\": container with ID starting with 07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53 not found: ID does not exist" containerID="07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53"
Oct 31 00:20:18.437269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.437193    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53"} err="failed to get container status \"07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53\": rpc error: code = NotFound desc = could not find container \"07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53\": container with ID starting with 07fd008faddca1defce9bec5d17f117754d8bd898b14e2581abd288fbc02fd53 not found: ID does not exist"
Oct 31 00:20:18.437269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.437221    5078 scope.go:122] "RemoveContainer" containerID="707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"
Oct 31 00:20:18.437574 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:20:18.437553    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29\": container with ID starting with 707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29 not found: ID does not exist" containerID="707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"
Oct 31 00:20:18.437654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.437583    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29"} err="failed to get container status \"707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29\": rpc error: code = NotFound desc = could not find container \"707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29\": container with ID starting with 707a30eee336658a24212c770a1b3c5caf3fd4951c3151b03e00a31ce5f83c29 not found: ID does not exist"
Oct 31 00:20:18.536772 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.536743    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5nzw\" (UniqueName: \"kubernetes.io/projected/c5986930-8d0d-4a08-8790-f49bcd74c771-kube-api-access-t5nzw\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:18.732036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.732003    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"]
Oct 31 00:20:18.738274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:18.738250    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3f561470-openshift-default-75989964cc-lgsqv"]
Oct 31 00:20:19.028991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.028910    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"]
Oct 31 00:20:19.034915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.034892    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6bd73d-qdsfv"]
Oct 31 00:20:19.979798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.979765    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"]
Oct 31 00:20:19.980587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980569    5078 cpu_manager.go:414] "containerMap: removing container" podUID="15c44c07-825b-43bf-a795-f1f32dd5019b" containerName="revoke-keys"
Oct 31 00:20:19.980587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980587    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c44c07-825b-43bf-a795-f1f32dd5019b" containerName="revoke-keys"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980614    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1fce4ea1-f092-421b-beee-1357420285ca" containerName="istio-proxy"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980620    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fce4ea1-f092-421b-beee-1357420285ca" containerName="istio-proxy"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980640    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7440c139-71fc-4ca0-a5a1-badd10be7aa4" containerName="payload-pre-processing"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980645    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7440c139-71fc-4ca0-a5a1-badd10be7aa4" containerName="payload-pre-processing"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980662    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980667    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980679    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy"
Oct 31 00:20:19.980723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980683    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy"
Oct 31 00:20:19.981109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980788    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy"
Oct 31 00:20:19.981109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980797    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" containerName="istio-proxy"
Oct 31 00:20:19.981109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980806    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c44c07-825b-43bf-a795-f1f32dd5019b" containerName="revoke-keys"
Oct 31 00:20:19.981109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980814    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fce4ea1-f092-421b-beee-1357420285ca" containerName="istio-proxy"
Oct 31 00:20:19.981109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.980823    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7440c139-71fc-4ca0-a5a1-badd10be7aa4" containerName="payload-pre-processing"
Oct 31 00:20:19.986656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:19.986634    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.030869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.030822    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f71e7a13-serving-cert\""
Oct 31 00:20:20.036378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.036354    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"]
Oct 31 00:20:20.061207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.060691    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.061207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.060848    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.061207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.060883    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.061207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.060999    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfbw\" (UniqueName: \"kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.163091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.163053    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.163440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.163409    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.163595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.163489    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.163707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.163687    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfbw\" (UniqueName: \"kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.164144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.164115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.167316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.167292    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.175072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.174992    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.178975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.178817    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfbw\" (UniqueName: \"kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw\") pod \"maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.215956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.211114    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c860a92-9fef-4d58-9a7a-74ccda427223" path="/var/lib/kubelet/pods/5c860a92-9fef-4d58-9a7a-74ccda427223/volumes"
Oct 31 00:20:20.215956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.211976    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5986930-8d0d-4a08-8790-f49bcd74c771" path="/var/lib/kubelet/pods/c5986930-8d0d-4a08-8790-f49bcd74c771/volumes"
Oct 31 00:20:20.303623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.303540    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:20.605659 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:20:20.605631    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396de4ab_7766_4d80_b934_98f9610a14c9.slice/crio-f0e8dae63252ef683e7786108221dd3d697fd53d64eb64a394e8684949d5a1c7 WatchSource:0}: Error finding container f0e8dae63252ef683e7786108221dd3d697fd53d64eb64a394e8684949d5a1c7: Status 404 returned error can't find the container with id f0e8dae63252ef683e7786108221dd3d697fd53d64eb64a394e8684949d5a1c7
Oct 31 00:20:20.607731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.607708    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:20:20.609285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:20.609264    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"]
Oct 31 00:20:21.428709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:21.428669    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" event={"ID":"396de4ab-7766-4d80-b934-98f9610a14c9","Type":"ContainerStarted","Data":"3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e"}
Oct 31 00:20:21.429141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:21.428738    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" event={"ID":"396de4ab-7766-4d80-b934-98f9610a14c9","Type":"ContainerStarted","Data":"f0e8dae63252ef683e7786108221dd3d697fd53d64eb64a394e8684949d5a1c7"}
Oct 31 00:20:21.429141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:21.428768    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:21.454961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:21.454900    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" podStartSLOduration=1.946092468 podStartE2EDuration="2.454886131s" totalImagesPullingTime="508.793663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:21.450757047 +0000 UTC m=+1589.397127765" watchObservedRunningTime="2026-10-31 00:20:21.454886131 +0000 UTC m=+1589.401256847"
Oct 31 00:20:23.028832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:23.028798    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"]
Oct 31 00:20:23.034555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:23.034533    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a546585-br47z"]
Oct 31 00:20:24.197145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:24.197112    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1baafdba-7832-4ece-a361-0c8dd68e54ae" path="/var/lib/kubelet/pods/1baafdba-7832-4ece-a361-0c8dd68e54ae/volumes"
Oct 31 00:20:27.035449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:27.035412    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"]
Oct 31 00:20:27.041853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:27.041823    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a428e1fe-6q2vg"]
Oct 31 00:20:27.447866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:27.447828    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:28.200830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:28.200798    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0271e86b-904a-47f2-80fb-79e51d10a378" path="/var/lib/kubelet/pods/0271e86b-904a-47f2-80fb-79e51d10a378/volumes"
Oct 31 00:20:30.033526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:30.033491    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"]
Oct 31 00:20:30.042047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:30.042010    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3371ee39-wmpzc"]
Oct 31 00:20:30.048480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:30.048442    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"]
Oct 31 00:20:30.071136 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:30.071087    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cd392b-l8cjb"]
Oct 31 00:20:30.199431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:30.199397    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb430dfd-014c-48cd-8d57-eab607347d3c" path="/var/lib/kubelet/pods/bb430dfd-014c-48cd-8d57-eab607347d3c/volumes"
Oct 31 00:20:30.199998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:30.199980    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f946d93d-fd1c-49d7-af04-77c709bfdd7d" path="/var/lib/kubelet/pods/f946d93d-fd1c-49d7-af04-77c709bfdd7d/volumes"
Oct 31 00:20:34.840037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.840004    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"]
Oct 31 00:20:34.903488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.895957    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:34.903773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.903747    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:20:34.903990 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.903944    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:20:34.914292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.914260    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"]
Oct 31 00:20:34.991939 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.991901    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:34.992114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.992061    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:34.992192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:34.992137    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2dq\" (UniqueName: \"kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.031582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.031550    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"]
Oct 31 00:20:35.038447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.038424    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"]
Oct 31 00:20:35.043705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.043687    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a043cfb7-cgvb2"]
Oct 31 00:20:35.048577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.048556    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"]
Oct 31 00:20:35.053414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.053396    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68b5313-vwwsg"]
Oct 31 00:20:35.058091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.058069    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6accf90-psqmw"]
Oct 31 00:20:35.094771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.094684    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.094926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.094807    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.094926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.094844    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2dq\" (UniqueName: \"kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.095991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.095965    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.098107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.098085    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.109284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.109257    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2dq\" (UniqueName: \"kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq\") pod \"maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.234613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.234583    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:35.534920 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:20:35.534894    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cd66f3_487a_4882_9ee0_596acaa670c4.slice/crio-a7286271203ac26292bb70262f0e62538cfafed7353f31c7e32c188730ec86ef WatchSource:0}: Error finding container a7286271203ac26292bb70262f0e62538cfafed7353f31c7e32c188730ec86ef: Status 404 returned error can't find the container with id a7286271203ac26292bb70262f0e62538cfafed7353f31c7e32c188730ec86ef
Oct 31 00:20:35.537196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:35.537176    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"]
Oct 31 00:20:36.202252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:36.202213    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a8d332-4698-4af0-b3c3-62b16b190adf" path="/var/lib/kubelet/pods/55a8d332-4698-4af0-b3c3-62b16b190adf/volumes"
Oct 31 00:20:36.202980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:36.202960    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e25ea5-d81f-4b94-9a70-877a9759262d" path="/var/lib/kubelet/pods/b7e25ea5-d81f-4b94-9a70-877a9759262d/volumes"
Oct 31 00:20:36.203591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:36.203574    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd23fec6-99e3-4b7a-a8e8-645abe23ff50" path="/var/lib/kubelet/pods/bd23fec6-99e3-4b7a-a8e8-645abe23ff50/volumes"
Oct 31 00:20:36.245369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:36.245333    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh" event={"ID":"22cd66f3-487a-4882-9ee0-596acaa670c4","Type":"ContainerStarted","Data":"e4bceaf7b3b5945523f6f9dd8e30e6993af19f8e8e15a57bdc9b828cb5b3d4ab"}
Oct 31 00:20:36.245623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:36.245396    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh" event={"ID":"22cd66f3-487a-4882-9ee0-596acaa670c4","Type":"ContainerStarted","Data":"a7286271203ac26292bb70262f0e62538cfafed7353f31c7e32c188730ec86ef"}
Oct 31 00:20:36.273323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:36.273266    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh" podStartSLOduration=2.273246209 podStartE2EDuration="2.273246209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:36.266681374 +0000 UTC m=+1604.213052091" watchObservedRunningTime="2026-10-31 00:20:36.273246209 +0000 UTC m=+1604.219616929"
Oct 31 00:20:36.410398 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:20:36.410357    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:20:37.257121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:37.257087    5078 generic.go:356] "Generic (PLEG): container finished" podID="22cd66f3-487a-4882-9ee0-596acaa670c4" containerID="e4bceaf7b3b5945523f6f9dd8e30e6993af19f8e8e15a57bdc9b828cb5b3d4ab" exitCode=0
Oct 31 00:20:37.257632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:37.257162    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh" event={"ID":"22cd66f3-487a-4882-9ee0-596acaa670c4","Type":"ContainerDied","Data":"e4bceaf7b3b5945523f6f9dd8e30e6993af19f8e8e15a57bdc9b828cb5b3d4ab"}
Oct 31 00:20:38.459103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.459065    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:38.559287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.559219    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca\") pod \"22cd66f3-487a-4882-9ee0-596acaa670c4\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") "
Oct 31 00:20:38.559444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.559429    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca\") pod \"22cd66f3-487a-4882-9ee0-596acaa670c4\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") "
Oct 31 00:20:38.559660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.559641    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq\" (UniqueName: \"kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq\") pod \"22cd66f3-487a-4882-9ee0-596acaa670c4\" (UID: \"22cd66f3-487a-4882-9ee0-596acaa670c4\") "
Oct 31 00:20:38.559767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.559642    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca" pod "22cd66f3-487a-4882-9ee0-596acaa670c4" (UID: "22cd66f3-487a-4882-9ee0-596acaa670c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:38.560257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.560237    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:38.561742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.561712    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca" pod "22cd66f3-487a-4882-9ee0-596acaa670c4" (UID: "22cd66f3-487a-4882-9ee0-596acaa670c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:38.561823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.561811    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq" pod "22cd66f3-487a-4882-9ee0-596acaa670c4" (UID: "22cd66f3-487a-4882-9ee0-596acaa670c4"). InnerVolumeSpecName "kube-api-access-4c2dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:38.661887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.661859    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2dq\" (UniqueName: \"kubernetes.io/projected/22cd66f3-487a-4882-9ee0-596acaa670c4-kube-api-access-4c2dq\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:38.661887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:38.661886    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22cd66f3-487a-4882-9ee0-596acaa670c4-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:39.281017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:39.280979    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh" event={"ID":"22cd66f3-487a-4882-9ee0-596acaa670c4","Type":"ContainerDied","Data":"a7286271203ac26292bb70262f0e62538cfafed7353f31c7e32c188730ec86ef"}
Oct 31 00:20:39.281017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:39.281005    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"
Oct 31 00:20:39.281286 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:39.281045    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7286271203ac26292bb70262f0e62538cfafed7353f31c7e32c188730ec86ef"
Oct 31 00:20:40.035518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:40.035478    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"]
Oct 31 00:20:40.041172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:40.041145    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945191b-p785v"]
Oct 31 00:20:40.200860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:40.200823    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8073975-c499-4cb6-ba86-9db9de1356eb" path="/var/lib/kubelet/pods/a8073975-c499-4cb6-ba86-9db9de1356eb/volumes"
Oct 31 00:20:43.033680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:43.033623    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"]
Oct 31 00:20:43.039719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:43.039689    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"]
Oct 31 00:20:43.045266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:43.045245    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf64215d-rfcdv"]
Oct 31 00:20:43.049874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:43.049842    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10c43b4-jnbwq"]
Oct 31 00:20:44.041487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:44.037708    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"]
Oct 31 00:20:44.052568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:44.052184    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74dd79e9-tph8k"]
Oct 31 00:20:44.197088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:44.197060    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7bdebb-1b97-4515-8ac9-724bbae4f6c0" path="/var/lib/kubelet/pods/2e7bdebb-1b97-4515-8ac9-724bbae4f6c0/volumes"
Oct 31 00:20:44.197548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:44.197533    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a168e155-83cf-4683-9e4e-a56681e0a24b" path="/var/lib/kubelet/pods/a168e155-83cf-4683-9e4e-a56681e0a24b/volumes"
Oct 31 00:20:44.197971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:44.197957    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a" path="/var/lib/kubelet/pods/ebf1b1fa-8e6a-42f1-bfb4-4a75bbaa0b4a/volumes"
Oct 31 00:20:52.427004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.426967    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"]
Oct 31 00:20:52.427557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.427248    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" podUID="396de4ab-7766-4d80-b934-98f9610a14c9" containerName="maas-api" containerID="cri-o://3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e" gracePeriod=30
Oct 31 00:20:52.446958 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.446912    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" podUID="396de4ab-7766-4d80-b934-98f9610a14c9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.156:8443/health\": dial tcp 10.132.0.156:8443: connect: connection refused"
Oct 31 00:20:52.710792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.710717    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"]
Oct 31 00:20:52.720068 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.719957    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f71e7a13-w9nlh"]
Oct 31 00:20:52.905490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.898647    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:52.909484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.906228    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw\" (UniqueName: \"kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw\") pod \"396de4ab-7766-4d80-b934-98f9610a14c9\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") "
Oct 31 00:20:52.909484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.906357    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca\") pod \"396de4ab-7766-4d80-b934-98f9610a14c9\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") "
Oct 31 00:20:52.909484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.906513    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls\") pod \"396de4ab-7766-4d80-b934-98f9610a14c9\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") "
Oct 31 00:20:52.909484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.906614    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs\") pod \"396de4ab-7766-4d80-b934-98f9610a14c9\" (UID: \"396de4ab-7766-4d80-b934-98f9610a14c9\") "
Oct 31 00:20:52.911254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.910947    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca" pod "396de4ab-7766-4d80-b934-98f9610a14c9" (UID: "396de4ab-7766-4d80-b934-98f9610a14c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:52.976728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:52.963590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls" pod "396de4ab-7766-4d80-b934-98f9610a14c9" (UID: "396de4ab-7766-4d80-b934-98f9610a14c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:53.065483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.064569    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396de4ab-7766-4d80-b934-98f9610a14c9-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:53.065483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.064606    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:53.153232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.149327    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs" pod "396de4ab-7766-4d80-b934-98f9610a14c9" (UID: "396de4ab-7766-4d80-b934-98f9610a14c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:53.153232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.149874    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw" pod "396de4ab-7766-4d80-b934-98f9610a14c9" (UID: "396de4ab-7766-4d80-b934-98f9610a14c9"). InnerVolumeSpecName "kube-api-access-9zfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:53.166827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.166794    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zfbw\" (UniqueName: \"kubernetes.io/projected/396de4ab-7766-4d80-b934-98f9610a14c9-kube-api-access-9zfbw\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:53.166827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.166829    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396de4ab-7766-4d80-b934-98f9610a14c9-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:53.533700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.533663    5078 generic.go:356] "Generic (PLEG): container finished" podID="396de4ab-7766-4d80-b934-98f9610a14c9" containerID="3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e" exitCode=0
Oct 31 00:20:53.534493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.534437    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" event={"ID":"396de4ab-7766-4d80-b934-98f9610a14c9","Type":"ContainerDied","Data":"3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e"}
Oct 31 00:20:53.534681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.534664    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt" event={"ID":"396de4ab-7766-4d80-b934-98f9610a14c9","Type":"ContainerDied","Data":"f0e8dae63252ef683e7786108221dd3d697fd53d64eb64a394e8684949d5a1c7"}
Oct 31 00:20:53.534799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.534786    5078 scope.go:122] "RemoveContainer" containerID="3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e"
Oct 31 00:20:53.536299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.536263    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"
Oct 31 00:20:53.592494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.568742    5078 scope.go:122] "RemoveContainer" containerID="3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e"
Oct 31 00:20:53.592494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.581744    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"]
Oct 31 00:20:53.606614 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:20:53.602708    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e\": container with ID starting with 3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e not found: ID does not exist" containerID="3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e"
Oct 31 00:20:53.606614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.602751    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e"} err="failed to get container status \"3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e\": rpc error: code = NotFound desc = could not find container \"3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e\": container with ID starting with 3f3c4f12ade204d11de642844ecce47b472f22eae780e636f27898cd506e642e not found: ID does not exist"
Oct 31 00:20:53.606836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:53.606649    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f71e7a13-5df4f4799-lq7lt"]
Oct 31 00:20:54.030651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.030622    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"]
Oct 31 00:20:54.046524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.045709    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b901e3-7ndxj"]
Oct 31 00:20:54.207495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.206129    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3f2de8-172d-4c9d-9e29-4ca873325531" path="/var/lib/kubelet/pods/0e3f2de8-172d-4c9d-9e29-4ca873325531/volumes"
Oct 31 00:20:54.207495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.206873    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cd66f3-487a-4882-9ee0-596acaa670c4" path="/var/lib/kubelet/pods/22cd66f3-487a-4882-9ee0-596acaa670c4/volumes"
Oct 31 00:20:54.207759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.207538    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396de4ab-7766-4d80-b934-98f9610a14c9" path="/var/lib/kubelet/pods/396de4ab-7766-4d80-b934-98f9610a14c9/volumes"
Oct 31 00:20:54.868364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.868330    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"]
Oct 31 00:20:54.869407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.869387    5078 cpu_manager.go:414] "containerMap: removing container" podUID="22cd66f3-487a-4882-9ee0-596acaa670c4" containerName="revoke-keys"
Oct 31 00:20:54.869407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.869407    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cd66f3-487a-4882-9ee0-596acaa670c4" containerName="revoke-keys"
Oct 31 00:20:54.879337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.869445    5078 cpu_manager.go:414] "containerMap: removing container" podUID="396de4ab-7766-4d80-b934-98f9610a14c9" containerName="maas-api"
Oct 31 00:20:54.879337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.869453    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396de4ab-7766-4d80-b934-98f9610a14c9" containerName="maas-api"
Oct 31 00:20:54.879337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.869655    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="396de4ab-7766-4d80-b934-98f9610a14c9" containerName="maas-api"
Oct 31 00:20:54.879337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.869687    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cd66f3-487a-4882-9ee0-596acaa670c4" containerName="revoke-keys"
Oct 31 00:20:54.882767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.882743    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.898853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.898384    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-41a0b7-openshift-default-dockercfg-f5qbq\""
Oct 31 00:20:54.900780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.900755    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"]
Oct 31 00:20:54.966492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966437    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.966735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966630    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.966735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966656    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.966874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966738    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.966874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966772    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.966874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.967039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966887    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.967039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.966950    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:54.967142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:54.967060    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.048746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.048711    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"]
Oct 31 00:20:55.056642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.056605    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a05f44-4pbv5"]
Oct 31 00:20:55.068476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068420    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068542    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068630    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068653    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068711    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068879    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.068959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.068922    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.069020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.069001    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.069075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.069019    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.069075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.069035    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.069244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.069217    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.069411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.069376    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.069620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.069598    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.071513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.071443    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.071513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.071457    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.087555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.087478    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.088505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.088446    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token\") pod \"e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") " pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.230485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.228347    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:55.720835 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:20:55.720796    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86073226_02be_4443_b886_654281e0a89a.slice/crio-7525b31ee764f02ad69c0c74f01675dbaff45b5380a97db4e75ab02565779b51 WatchSource:0}: Error finding container 7525b31ee764f02ad69c0c74f01675dbaff45b5380a97db4e75ab02565779b51: Status 404 returned error can't find the container with id 7525b31ee764f02ad69c0c74f01675dbaff45b5380a97db4e75ab02565779b51
Oct 31 00:20:55.721997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.721975    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"]
Oct 31 00:20:55.723602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.723570    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:20:55.723670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.723638    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:20:55.723670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:55.723666    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:20:56.226161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:56.226125    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45db12f8-975f-45c3-a7e1-064f7135d6e1" path="/var/lib/kubelet/pods/45db12f8-975f-45c3-a7e1-064f7135d6e1/volumes"
Oct 31 00:20:56.669636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:56.669582    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" event={"ID":"86073226-02be-4443-b886-654281e0a89a","Type":"ContainerStarted","Data":"e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"}
Oct 31 00:20:56.669844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:56.669657    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" event={"ID":"86073226-02be-4443-b886-654281e0a89a","Type":"ContainerStarted","Data":"7525b31ee764f02ad69c0c74f01675dbaff45b5380a97db4e75ab02565779b51"}
Oct 31 00:20:56.701821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:56.701766    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podStartSLOduration=2.701749884 podStartE2EDuration="2.701749884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:56.696644944 +0000 UTC m=+1624.643015700" watchObservedRunningTime="2026-10-31 00:20:56.701749884 +0000 UTC m=+1624.648120601"
Oct 31 00:20:57.229620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:57.229588    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:57.235534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:57.235508    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:57.684606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:57.684455    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:57.686535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:57.686439    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:20:58.797176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.797136    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"]
Oct 31 00:20:58.803004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.802971    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.810857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.810838    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:20:58.813626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.813602    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"]
Oct 31 00:20:58.822781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.822757    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:20:58.850909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.850875    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.850909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.850914    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.851130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.851073    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7dg\" (UniqueName: \"kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.953758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.953723    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.954048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.954007    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.954200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.954178    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7dg\" (UniqueName: \"kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.954655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.954628    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.956876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.956848    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:58.968632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:58.968608    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7dg\" (UniqueName: \"kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg\") pod \"maas-api-revoke-sub-21935661-2fzc7\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:59.131500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:59.131400    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:20:59.454565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:59.452507    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"]
Oct 31 00:20:59.859492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:20:59.831481    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7" event={"ID":"26289741-7ab5-4810-8edf-dd77dfb9663d","Type":"ContainerStarted","Data":"eedccf02bdc28e2b02c38f27efca4a26dd3de05631d98c1d9969c470c009b5c3"}
Oct 31 00:21:00.843088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:00.843056    5078 generic.go:356] "Generic (PLEG): container finished" podID="26289741-7ab5-4810-8edf-dd77dfb9663d" containerID="df1304ea316ce05791751fed5c6ec9eb298e602c88d5f94b5da50d22e771736e" exitCode=0
Oct 31 00:21:00.843259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:00.843136    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7" event={"ID":"26289741-7ab5-4810-8edf-dd77dfb9663d","Type":"ContainerDied","Data":"df1304ea316ce05791751fed5c6ec9eb298e602c88d5f94b5da50d22e771736e"}
Oct 31 00:21:02.314631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.314594    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:21:02.385610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.385571    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca\") pod \"26289741-7ab5-4810-8edf-dd77dfb9663d\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") "
Oct 31 00:21:02.385974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.385950    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca\") pod \"26289741-7ab5-4810-8edf-dd77dfb9663d\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") "
Oct 31 00:21:02.386085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.386072    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg\" (UniqueName: \"kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg\") pod \"26289741-7ab5-4810-8edf-dd77dfb9663d\" (UID: \"26289741-7ab5-4810-8edf-dd77dfb9663d\") "
Oct 31 00:21:02.386901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.386874    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca" pod "26289741-7ab5-4810-8edf-dd77dfb9663d" (UID: "26289741-7ab5-4810-8edf-dd77dfb9663d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:21:02.411244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.411205    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg" pod "26289741-7ab5-4810-8edf-dd77dfb9663d" (UID: "26289741-7ab5-4810-8edf-dd77dfb9663d"). InnerVolumeSpecName "kube-api-access-4b7dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:02.414567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.414535    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca" pod "26289741-7ab5-4810-8edf-dd77dfb9663d" (UID: "26289741-7ab5-4810-8edf-dd77dfb9663d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:02.488991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.488905    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:02.488991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.488949    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b7dg\" (UniqueName: \"kubernetes.io/projected/26289741-7ab5-4810-8edf-dd77dfb9663d-kube-api-access-4b7dg\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:02.488991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.488968    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26289741-7ab5-4810-8edf-dd77dfb9663d-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:02.998490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.997590    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7" event={"ID":"26289741-7ab5-4810-8edf-dd77dfb9663d","Type":"ContainerDied","Data":"eedccf02bdc28e2b02c38f27efca4a26dd3de05631d98c1d9969c470c009b5c3"}
Oct 31 00:21:02.998490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.997658    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eedccf02bdc28e2b02c38f27efca4a26dd3de05631d98c1d9969c470c009b5c3"
Oct 31 00:21:02.998490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:02.997732    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"
Oct 31 00:21:03.699480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.699436    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"]
Oct 31 00:21:03.700303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.700286    5078 cpu_manager.go:414] "containerMap: removing container" podUID="26289741-7ab5-4810-8edf-dd77dfb9663d" containerName="revoke-keys"
Oct 31 00:21:03.700388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.700306    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26289741-7ab5-4810-8edf-dd77dfb9663d" containerName="revoke-keys"
Oct 31 00:21:03.700492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.700481    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="26289741-7ab5-4810-8edf-dd77dfb9663d" containerName="revoke-keys"
Oct 31 00:21:03.704514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.704493    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:03.716419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.716391    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-14e39b\""
Oct 31 00:21:03.716706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.716689    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-14e39b-dockercfg-2gndj\""
Oct 31 00:21:03.724040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.723239    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"]
Oct 31 00:21:03.829598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.813802    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:03.829598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.813945    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98j48\" (UniqueName: \"kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48\") pod \"payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:03.939495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.937449    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98j48\" (UniqueName: \"kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48\") pod \"payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:03.939495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.937857    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:03.949535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:03.941327    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:04.069546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:04.069487    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98j48\" (UniqueName: \"kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48\") pod \"payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:04.155085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:04.155051    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:04.489443 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:21:04.489391    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2451ffa2_2e2c_4f9e_a731_e6969efc1e4c.slice/crio-b776ed84452a76fbbde7c7db845fa57f187fd8e63983357b032b58a55add1983 WatchSource:0}: Error finding container b776ed84452a76fbbde7c7db845fa57f187fd8e63983357b032b58a55add1983: Status 404 returned error can't find the container with id b776ed84452a76fbbde7c7db845fa57f187fd8e63983357b032b58a55add1983
Oct 31 00:21:04.492164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:04.492139    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"]
Oct 31 00:21:05.065830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:05.065801    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" event={"ID":"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c","Type":"ContainerStarted","Data":"90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c"}
Oct 31 00:21:05.066275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:05.065851    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" event={"ID":"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c","Type":"ContainerStarted","Data":"b776ed84452a76fbbde7c7db845fa57f187fd8e63983357b032b58a55add1983"}
Oct 31 00:21:05.066275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:05.065898    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:05.143595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:05.143540    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podStartSLOduration=2.143523578 podStartE2EDuration="2.143523578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:05.123929375 +0000 UTC m=+1633.070300094" watchObservedRunningTime="2026-10-31 00:21:05.143523578 +0000 UTC m=+1633.089894294"
Oct 31 00:21:06.454803 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:21:06.454769    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:21:11.886486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:11.884714    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"]
Oct 31 00:21:11.895488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:11.891150    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:11.911514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:11.909711    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-41a0b7-serving-cert\""
Oct 31 00:21:11.911514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:11.910420    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"]
Oct 31 00:21:12.032224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.032187    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.032413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.032231    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxhr\" (UniqueName: \"kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.032413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.032307    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.032639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.032612    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.135794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.135754    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.136593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.136526    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxhr\" (UniqueName: \"kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.136820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.136800    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.137639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.137615    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.138259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.138233    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.146318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.146289    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.147399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.147369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.150078 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.150052    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxhr\" (UniqueName: \"kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr\") pod \"maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.228000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.227972    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:12.596433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:12.593746    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"]
Oct 31 00:21:13.197402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:13.197303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" event={"ID":"010f4cfc-84d4-4b18-a9ff-289488077e9b","Type":"ContainerStarted","Data":"5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b"}
Oct 31 00:21:13.197402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:13.197366    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" event={"ID":"010f4cfc-84d4-4b18-a9ff-289488077e9b","Type":"ContainerStarted","Data":"dcabb5fa933d7729e8719bf6622bfd6ed5b4feaca6a94a5f88cd084362a52097"}
Oct 31 00:21:13.198450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:13.198205    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:13.243980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:13.243910    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podStartSLOduration=1.748478466 podStartE2EDuration="2.243891451s" totalImagesPullingTime="495.412985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:13.242888143 +0000 UTC m=+1641.189258861" watchObservedRunningTime="2026-10-31 00:21:13.243891451 +0000 UTC m=+1641.190262172"
Oct 31 00:21:16.089513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:16.087369    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:21:19.218041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:19.218001    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:21:24.236197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.234562    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"]
Oct 31 00:21:24.249852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.249811    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"]
Oct 31 00:21:24.250058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.249934    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.327484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.326831    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-7a7688-openshift-default-dockercfg-c2t4t\""
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.436759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkb5b\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.436894    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.436936    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.436966    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.437032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.437069    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.437485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.437154    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.438032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.437529    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.438032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.437558    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.543792    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.543843    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.543937    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkb5b\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.544029    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.544071    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.544100    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.544164    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.544202    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.544538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.544300    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.557217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.548684    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.557217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.549345    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.557217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.555746    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.557217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.556543    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.557217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.556889    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.614001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.612755    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.614001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.613274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.614564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.614319    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkb5b\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.625495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.625379    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:24.903286 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:24.900926    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:25.373232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.373206    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"]
Oct 31 00:21:25.374271 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:21:25.374247    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86417f75_420a_4ef6_84ee_1d36bd8632ca.slice/crio-6a1654258341307c9004b3aa76426c1c2c80c0e7370ea75b683132f8466e4c75 WatchSource:0}: Error finding container 6a1654258341307c9004b3aa76426c1c2c80c0e7370ea75b683132f8466e4c75: Status 404 returned error can't find the container with id 6a1654258341307c9004b3aa76426c1c2c80c0e7370ea75b683132f8466e4c75
Oct 31 00:21:25.377445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.377407    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:25.377578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.377526    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:25.377578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.377561    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:25.613486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.610483    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" event={"ID":"86417f75-420a-4ef6-84ee-1d36bd8632ca","Type":"ContainerStarted","Data":"5a155a3b85f1461cac00c73db5137ff20b569a162b1005b5426884a75e779b2f"}
Oct 31 00:21:25.613486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.610554    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" event={"ID":"86417f75-420a-4ef6-84ee-1d36bd8632ca","Type":"ContainerStarted","Data":"6a1654258341307c9004b3aa76426c1c2c80c0e7370ea75b683132f8466e4c75"}
Oct 31 00:21:25.651486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.648334    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podStartSLOduration=1.6483156509999999 podStartE2EDuration="1.648315651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:25.644970495 +0000 UTC m=+1653.591341213" watchObservedRunningTime="2026-10-31 00:21:25.648315651 +0000 UTC m=+1653.594686370"
Oct 31 00:21:25.906515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:25.906408    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:26.917485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:26.916893    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:27.647488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:27.646933    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:27.657529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:27.654697    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:21:29.697486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.697287    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"]
Oct 31 00:21:29.727501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.725831    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:29.733493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.731822    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"]
Oct 31 00:21:29.738069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.737233    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-7a7688\""
Oct 31 00:21:29.741677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.741655    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"]
Oct 31 00:21:29.741850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.741764    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:29.751506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.751333    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"]
Oct 31 00:21:29.757152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.753833    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-7a7688-dockercfg-2w6bc\""
Oct 31 00:21:29.872495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.846522    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnwq2\" (UniqueName: \"kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2\") pod \"payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:29.894336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.894293    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvrj\" (UniqueName: \"kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj\") pod \"payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:29.909109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.909069    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:29.909565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:29.909521    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:30.049548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.043965    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:30.049548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.044022    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:30.049548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.044123    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwq2\" (UniqueName: \"kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2\") pod \"payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:30.049548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.044320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvrj\" (UniqueName: \"kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj\") pod \"payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:30.049548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.045748    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:30.049548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.046619    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:30.111357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.111310    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwq2\" (UniqueName: \"kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2\") pod \"payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:30.112581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.112545    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvrj\" (UniqueName: \"kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj\") pod \"payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:30.359692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.359611    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:30.370127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.370099    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:30.439057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.435834    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"]
Oct 31 00:21:30.475758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.475737    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.490480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.490436    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"]
Oct 31 00:21:30.501545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.500964    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a39be4-openshift-default-dockercfg-clxb8\""
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.567732    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrz4r\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.567792    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.567895    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.567945    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.568053    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.568129    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.568158    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.568232    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.632550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.568270    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.722486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.710619    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.723102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.723074    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.735489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.723439    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrz4r\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.763435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.763369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.763954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.718941    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.824843    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.825059    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.825130    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.825272    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.825367    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.825391    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.826220    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.826776    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.826991    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.892683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.827186    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.917238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.917203    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.918408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.918356    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrz4r\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:30.925232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:30.925183    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:31.125589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.124305    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:31.158256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.158213    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"]
Oct 31 00:21:31.201352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.181231    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"]
Oct 31 00:21:31.524592 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:21:31.524558    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f1a149_d6c7_4c5f_97c1_639866c3b6f5.slice/crio-56798db62f88b7b12f1cc236f2d2ad4c45639d226b2b7d03382b4181db18b957 WatchSource:0}: Error finding container 56798db62f88b7b12f1cc236f2d2ad4c45639d226b2b7d03382b4181db18b957: Status 404 returned error can't find the container with id 56798db62f88b7b12f1cc236f2d2ad4c45639d226b2b7d03382b4181db18b957
Oct 31 00:21:31.528171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.528137    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:31.528278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.528203    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:31.528278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.528236    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:31.531778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.531756    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"]
Oct 31 00:21:31.920540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.920504    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" event={"ID":"325f8eed-03c3-4f20-a72a-7753e560cb45","Type":"ContainerStarted","Data":"de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8"}
Oct 31 00:21:31.921139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.920569    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" event={"ID":"325f8eed-03c3-4f20-a72a-7753e560cb45","Type":"ContainerStarted","Data":"bd838dce2f58387c9e899064122e377a501788cf0a5bfeaded1bdd2d03f49b3d"}
Oct 31 00:21:31.921139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.920691    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:31.922801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.922771    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" event={"ID":"a75e181d-6850-4b90-a34a-3a1d098a6f53","Type":"ContainerStarted","Data":"bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed"}
Oct 31 00:21:31.922997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.922979    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" event={"ID":"a75e181d-6850-4b90-a34a-3a1d098a6f53","Type":"ContainerStarted","Data":"1aa6b03a0a507a5333aadfeac6d0c12911a98e642e830b0781bf7e93691322b8"}
Oct 31 00:21:31.923120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.923107    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:31.924971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.924940    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" event={"ID":"22f1a149-d6c7-4c5f-97c1-639866c3b6f5","Type":"ContainerStarted","Data":"9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"}
Oct 31 00:21:31.925088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.924995    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" event={"ID":"22f1a149-d6c7-4c5f-97c1-639866c3b6f5","Type":"ContainerStarted","Data":"56798db62f88b7b12f1cc236f2d2ad4c45639d226b2b7d03382b4181db18b957"}
Oct 31 00:21:31.950065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:31.950007    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" podStartSLOduration=2.949992129 podStartE2EDuration="2.949992129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:31.946863414 +0000 UTC m=+1659.893234159" watchObservedRunningTime="2026-10-31 00:21:31.949992129 +0000 UTC m=+1659.896362846"
Oct 31 00:21:32.006506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:32.006439    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" podStartSLOduration=3.006425244 podStartE2EDuration="3.006425244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:32.004550718 +0000 UTC m=+1659.950921436" watchObservedRunningTime="2026-10-31 00:21:32.006425244 +0000 UTC m=+1659.952795957"
Oct 31 00:21:32.009516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:32.009452    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podStartSLOduration=2.009438777 podStartE2EDuration="2.009438777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:31.981928067 +0000 UTC m=+1659.928298804" watchObservedRunningTime="2026-10-31 00:21:32.009438777 +0000 UTC m=+1659.955809493"
Oct 31 00:21:32.124532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:32.124502    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:32.129838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:32.129812    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:32.936538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:32.936506    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:32.937423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:32.937405    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:21:36.004494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.002684    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"]
Oct 31 00:21:36.021420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.021387    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"]
Oct 31 00:21:36.021678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.021545    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.029699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.029664    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a39be4-dockercfg-dzch5\""
Oct 31 00:21:36.076394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.076213    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a39be4\""
Oct 31 00:21:36.101414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.101384    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.101862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.101833    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5m2\" (UniqueName: \"kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2\") pod \"payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.244482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.239015    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.244482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.239085    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5m2\" (UniqueName: \"kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2\") pod \"payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.244482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.240305    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.564267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.563700    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5m2\" (UniqueName: \"kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2\") pod \"payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.612568 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:21:36.612527    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:21:36.643641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.643605    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:36.953920 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:21:36.953889    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4de92e_ac35_496d_9c11_297373884443.slice/crio-844e0ace63306c759dc814b421441bdc35991a53f2b4a85b72ae820ce2a38836 WatchSource:0}: Error finding container 844e0ace63306c759dc814b421441bdc35991a53f2b4a85b72ae820ce2a38836: Status 404 returned error can't find the container with id 844e0ace63306c759dc814b421441bdc35991a53f2b4a85b72ae820ce2a38836
Oct 31 00:21:36.955664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:36.955600    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"]
Oct 31 00:21:37.113924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:37.113885    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" event={"ID":"cc4de92e-ac35-496d-9c11-297373884443","Type":"ContainerStarted","Data":"0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b"}
Oct 31 00:21:37.114407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:37.113953    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" event={"ID":"cc4de92e-ac35-496d-9c11-297373884443","Type":"ContainerStarted","Data":"844e0ace63306c759dc814b421441bdc35991a53f2b4a85b72ae820ce2a38836"}
Oct 31 00:21:37.114407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:37.113986    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:37.142699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:37.142652    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podStartSLOduration=2.142638869 podStartE2EDuration="2.142638869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:37.138305698 +0000 UTC m=+1665.084676414" watchObservedRunningTime="2026-10-31 00:21:37.142638869 +0000 UTC m=+1665.089009643"
Oct 31 00:21:39.581361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.581323    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"]
Oct 31 00:21:39.601347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.601321    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.609679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.608571    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0df3f8-openshift-default-dockercfg-dw7vf\""
Oct 31 00:21:39.623851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.623824    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"]
Oct 31 00:21:39.632650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.632048    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.637669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.637637    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"]
Oct 31 00:21:39.649027 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.649011    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"]
Oct 31 00:21:39.701531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.701492    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.701692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.701610    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.701692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.701679    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.701791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.701775    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.701886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.701865    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.701951 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.701915    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.702052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.702028    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl628\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.702139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.702125    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.702202 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.702187    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.804951 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.804916    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.804978    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805019    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805079    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805161    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805207    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805231    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805260    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805287    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805313    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805389    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805427    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805445    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805449    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805631    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.805740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805672    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl628\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805751    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805793    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.805998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.805824    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.806190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.806086    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.806349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.806325    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.806506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.806363    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.806795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.806762    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.809519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.809449    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.810140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.810117    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.822511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.822484    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.822685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.822665    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl628\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.907775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.907744    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.907935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.907853    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.907981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.907944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.907981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.907973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908058    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908091    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908214 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908192    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908275    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908307    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908364    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908697    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908745    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.908808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.908752    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.910931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.910902    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.911088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.911066    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.921917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.921897    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.922626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.922599    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:39.929262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.929241    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:39.954290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:39.954252    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:40.255230 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:21:40.255203    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba615fc_186e_48cb_9bf1_18ee9c33dc9a.slice/crio-7dea302e3457b0493d6e004620fd56fab7bb14bd746ac46f35f6d1eb1a99a2d4 WatchSource:0}: Error finding container 7dea302e3457b0493d6e004620fd56fab7bb14bd746ac46f35f6d1eb1a99a2d4: Status 404 returned error can't find the container with id 7dea302e3457b0493d6e004620fd56fab7bb14bd746ac46f35f6d1eb1a99a2d4
Oct 31 00:21:40.258544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.258430    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:40.258544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.258537    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:40.258723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.258588    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:40.258815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.258792    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"]
Oct 31 00:21:40.284215 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:21:40.284181    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b7a6bf_9d4d_4c36_ae14_19a2bf4d0377.slice/crio-967f76e8003e5f8601bb1ab9938fca81aa8b43b8f58d1ec589d6f024f9ea7862 WatchSource:0}: Error finding container 967f76e8003e5f8601bb1ab9938fca81aa8b43b8f58d1ec589d6f024f9ea7862: Status 404 returned error can't find the container with id 967f76e8003e5f8601bb1ab9938fca81aa8b43b8f58d1ec589d6f024f9ea7862
Oct 31 00:21:40.287636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.287600    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:40.287737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.287721    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:40.287795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.287771    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:21:40.288933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:40.288913    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"]
Oct 31 00:21:41.163076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.162051    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" event={"ID":"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377","Type":"ContainerStarted","Data":"2eb948f8ec0d61796cf275d27a9c21e292c4ef3dd00beb8674c6aabcbfa4cc7d"}
Oct 31 00:21:41.163076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.162116    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" event={"ID":"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377","Type":"ContainerStarted","Data":"967f76e8003e5f8601bb1ab9938fca81aa8b43b8f58d1ec589d6f024f9ea7862"}
Oct 31 00:21:41.164883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.164854    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" event={"ID":"bba615fc-186e-48cb-9bf1-18ee9c33dc9a","Type":"ContainerStarted","Data":"7bbb4b7931d9b1edb29049aa378dd4e6dc96d69f7432afc53a7cfb7a810d25e1"}
Oct 31 00:21:41.165034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.164907    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" event={"ID":"bba615fc-186e-48cb-9bf1-18ee9c33dc9a","Type":"ContainerStarted","Data":"7dea302e3457b0493d6e004620fd56fab7bb14bd746ac46f35f6d1eb1a99a2d4"}
Oct 31 00:21:41.351746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.351554    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podStartSLOduration=2.351535501 podStartE2EDuration="2.351535501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:41.332144876 +0000 UTC m=+1669.278515605" watchObservedRunningTime="2026-10-31 00:21:41.351535501 +0000 UTC m=+1669.297906219"
Oct 31 00:21:41.363516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.359118    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podStartSLOduration=2.35910455 podStartE2EDuration="2.35910455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:41.300975589 +0000 UTC m=+1669.247346306" watchObservedRunningTime="2026-10-31 00:21:41.35910455 +0000 UTC m=+1669.305475268"
Oct 31 00:21:41.930619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.930581    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:41.931785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.931663    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 00:21:41.931785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.931744    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 00:21:41.955250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.955227    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:41.956305 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.956281    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 00:21:41.956369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:41.956349    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 00:21:42.934613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:42.930681    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 00:21:42.934613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:42.930753    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 00:21:42.939998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:42.939973    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:21:42.940162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:42.940073    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:21:42.955557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:42.955528    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 00:21:42.955714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:42.955580    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 00:21:43.931942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:43.931903    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 00:21:43.935487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:43.932223    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 00:21:43.962795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:43.962763    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 00:21:43.963060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:43.963032    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 00:21:44.932034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:44.931989    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 00:21:44.950689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:44.935568    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 00:21:44.959369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:44.959332    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 00:21:44.990694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:44.990548    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 00:21:45.929897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:45.929857    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 00:21:45.930220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:45.930188    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 00:21:45.955084 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:45.955053    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 00:21:45.955599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:45.955098    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 00:21:46.930102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:46.930059    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 00:21:46.930339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:46.930149    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 00:21:46.955738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:46.955703    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 00:21:46.956203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:46.955767    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 00:21:47.934975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:47.934945    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:47.935434 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:47.935403    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:47.936230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:47.936208    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:21:47.959058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:47.959040    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:47.959438 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:47.959310    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:47.960104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:47.960089    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:21:48.127310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:48.127278    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:21:49.047783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:49.047746    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"]
Oct 31 00:21:49.057313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:49.057277    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9d5bbe-m2d7m"]
Oct 31 00:21:50.199859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:50.199818    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656b98cc-a8be-402e-ac4e-a90ec2f12c38" path="/var/lib/kubelet/pods/656b98cc-a8be-402e-ac4e-a90ec2f12c38/volumes"
Oct 31 00:21:53.818355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.818298    5078 scope.go:122] "RemoveContainer" containerID="e1416a54a84c615dccb9a0d1c36b533dda18aec2a6b9197118d2f2948b192d5f"
Oct 31 00:21:53.833801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.833776    5078 scope.go:122] "RemoveContainer" containerID="97f1f07c9013a57c52cf16322b0e3d3e8aae9ed7679a99ab01006c20d5902efb"
Oct 31 00:21:53.846055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.846024    5078 scope.go:122] "RemoveContainer" containerID="c801034770ddf9c68bf42feb3043a59fdac295ebfbf86c17071fffeb01cbe4fa"
Oct 31 00:21:53.858649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.858614    5078 scope.go:122] "RemoveContainer" containerID="097eb5ca201caafabdbd9293102aa425761ef28ee5db223722c07bea9e4b8fbd"
Oct 31 00:21:53.870523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.870505    5078 scope.go:122] "RemoveContainer" containerID="a186bbbe9dc4af77601d5f7ed33819a0ae4af69b8f6245f91218d49dd2066560"
Oct 31 00:21:53.882704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.882677    5078 scope.go:122] "RemoveContainer" containerID="6c98564594d3ef00f7b1df8bb6348e420f93af33766de7c20e652cc5bd1c3ad5"
Oct 31 00:21:53.894356 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.894339    5078 scope.go:122] "RemoveContainer" containerID="c65ec3e175a483513a7c89c9f78727cc473ead4f0980d5cb2a721743bdf731d8"
Oct 31 00:21:53.953367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.953332    5078 scope.go:122] "RemoveContainer" containerID="6d9984a7acf26001621ec0d44b039c375cb9e3f4c941b8ec3becba2da21f6deb"
Oct 31 00:21:53.989069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:53.985353    5078 scope.go:122] "RemoveContainer" containerID="e340cd9a484d5f1a3e8ff858533d197d2a6422eb7f6b49bd94f30c7a974f6d48"
Oct 31 00:21:54.003218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.003079    5078 scope.go:122] "RemoveContainer" containerID="8acb12f34b39bd50886709b2e9c8425cc3552f916bb1efafcf09905cbe6827e6"
Oct 31 00:21:54.029706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.029679    5078 scope.go:122] "RemoveContainer" containerID="c405091f1fbb92f38b9e47343023b907646e1896fc7a12ce796cbe8ee2073dd7"
Oct 31 00:21:54.048507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.048480    5078 scope.go:122] "RemoveContainer" containerID="c290c619a5f1910efdaa33c233bb3c046a6b7d1da8ae60aaf6a8eb1664d11149"
Oct 31 00:21:54.063619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.063596    5078 scope.go:122] "RemoveContainer" containerID="93430db61b1b058db139d6fd089c3896f4a1c11bbc786766de9f15f76740a0b5"
Oct 31 00:21:54.089943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.089916    5078 scope.go:122] "RemoveContainer" containerID="62e2d58217835bd6c7e1d61b7efc306bd0c133c4a94aac60b6c063da7d356da7"
Oct 31 00:21:54.110130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.110108    5078 scope.go:122] "RemoveContainer" containerID="4ccb4a61c84ddad894325c1094774773b05700bd68c70bf4618aa4c2ebafe4c2"
Oct 31 00:21:54.122053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:54.122025    5078 scope.go:122] "RemoveContainer" containerID="d9937420a2d46c2985a562ad0eb0d974743b754314359e0e8a5943220d14ac3d"
Oct 31 00:21:56.032010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:56.031971    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"]
Oct 31 00:21:56.038059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:56.038026    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426e0fd-8pdsx"]
Oct 31 00:21:56.197487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:21:56.197443    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28464d3c-03c3-4201-b085-2d9406b99c36" path="/var/lib/kubelet/pods/28464d3c-03c3-4201-b085-2d9406b99c36/volumes"
Oct 31 00:22:03.074557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.074518    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"]
Oct 31 00:22:03.170489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.145788    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf200d6-k2prv"]
Oct 31 00:22:03.267622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.267572    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"]
Oct 31 00:22:03.343259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.341819    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"]
Oct 31 00:22:03.343259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.341957    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.348380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.348312    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:22:03.348567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.348385    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:22:03.406488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.403816    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.406488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.403905    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.406488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.404067    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r5h\" (UniqueName: \"kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.509488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.508624    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.509488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.508763    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r5h\" (UniqueName: \"kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.509488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.509053    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.511492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.509948    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.529952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.529920    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.530946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.530921    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r5h\" (UniqueName: \"kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h\") pod \"maas-api-revoke-sub-260cf387-7t7th\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:03.690814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:03.690771    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:04.290543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:04.269525    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d08871-b160-46b6-a606-91638cdfcd52" path="/var/lib/kubelet/pods/c9d08871-b160-46b6-a606-91638cdfcd52/volumes"
Oct 31 00:22:04.625540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:04.607876    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"]
Oct 31 00:22:04.959244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:04.959208    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th" event={"ID":"8578bae2-cc5d-40a6-87b2-19c129319314","Type":"ContainerStarted","Data":"25c05ee0bc8e5a78e0ef05ab026201483672ea89367b7e7de27bf160b11342cf"}
Oct 31 00:22:06.049081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:06.048983    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th" event={"ID":"8578bae2-cc5d-40a6-87b2-19c129319314","Type":"ContainerStarted","Data":"f02aeb372dce85181e752422eed1fc994b30e3aa0e0aba76a9b72aaabc09cb5f"}
Oct 31 00:22:06.133486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:06.132614    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th" podStartSLOduration=3.132589081 podStartE2EDuration="3.132589081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:06.079154534 +0000 UTC m=+1694.025525253" watchObservedRunningTime="2026-10-31 00:22:06.132589081 +0000 UTC m=+1694.078959800"
Oct 31 00:22:06.549648 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:22:06.549512    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:22:07.054441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:07.054354    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"]
Oct 31 00:22:07.068257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:07.068217    5078 generic.go:356] "Generic (PLEG): container finished" podID="8578bae2-cc5d-40a6-87b2-19c129319314" containerID="f02aeb372dce85181e752422eed1fc994b30e3aa0e0aba76a9b72aaabc09cb5f" exitCode=0
Oct 31 00:22:07.076734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:07.076696    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th" event={"ID":"8578bae2-cc5d-40a6-87b2-19c129319314","Type":"ContainerDied","Data":"f02aeb372dce85181e752422eed1fc994b30e3aa0e0aba76a9b72aaabc09cb5f"}
Oct 31 00:22:07.142273 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:07.142238    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fb7049-69vkv"]
Oct 31 00:22:08.229032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.228958    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cd632a-5414-465b-9406-b94e59152783" path="/var/lib/kubelet/pods/70cd632a-5414-465b-9406-b94e59152783/volumes"
Oct 31 00:22:08.341177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.341103    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:08.451994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.451965    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca\") pod \"8578bae2-cc5d-40a6-87b2-19c129319314\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") "
Oct 31 00:22:08.452190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.452052    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h\" (UniqueName: \"kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h\") pod \"8578bae2-cc5d-40a6-87b2-19c129319314\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") "
Oct 31 00:22:08.452190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.452131    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca\") pod \"8578bae2-cc5d-40a6-87b2-19c129319314\" (UID: \"8578bae2-cc5d-40a6-87b2-19c129319314\") "
Oct 31 00:22:08.452398 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.452375    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca" pod "8578bae2-cc5d-40a6-87b2-19c129319314" (UID: "8578bae2-cc5d-40a6-87b2-19c129319314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:08.452886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.452866    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:08.456223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.456195    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca" pod "8578bae2-cc5d-40a6-87b2-19c129319314" (UID: "8578bae2-cc5d-40a6-87b2-19c129319314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:08.456566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.456548    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h" pod "8578bae2-cc5d-40a6-87b2-19c129319314" (UID: "8578bae2-cc5d-40a6-87b2-19c129319314"). InnerVolumeSpecName "kube-api-access-56r5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:08.554495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.554447    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56r5h\" (UniqueName: \"kubernetes.io/projected/8578bae2-cc5d-40a6-87b2-19c129319314-kube-api-access-56r5h\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:08.554495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.554493    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8578bae2-cc5d-40a6-87b2-19c129319314-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:08.736077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.736044    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"]
Oct 31 00:22:08.737382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.737353    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8578bae2-cc5d-40a6-87b2-19c129319314" containerName="revoke-keys"
Oct 31 00:22:08.737382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.737381    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8578bae2-cc5d-40a6-87b2-19c129319314" containerName="revoke-keys"
Oct 31 00:22:08.737652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.737637    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8578bae2-cc5d-40a6-87b2-19c129319314" containerName="revoke-keys"
Oct 31 00:22:08.750487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.748260    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.754701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.754666    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"]
Oct 31 00:22:08.861800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.861766    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kbt\" (UniqueName: \"kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.862069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.862037    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.862319 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.862295    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.963790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.963758    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kbt\" (UniqueName: \"kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.963992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.963940    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.964065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.964005    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.964712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.964691    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.966704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.966681    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:08.977804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:08.977782    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kbt\" (UniqueName: \"kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt\") pod \"maas-api-revoke-sub-af86ada3-z6glk\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:09.103359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.103253    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th" event={"ID":"8578bae2-cc5d-40a6-87b2-19c129319314","Type":"ContainerDied","Data":"25c05ee0bc8e5a78e0ef05ab026201483672ea89367b7e7de27bf160b11342cf"}
Oct 31 00:22:09.103588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.103363    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c05ee0bc8e5a78e0ef05ab026201483672ea89367b7e7de27bf160b11342cf"
Oct 31 00:22:09.103588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.103437    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"
Oct 31 00:22:09.127405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.127378    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:22:09.410585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.396478    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"]
Oct 31 00:22:09.425409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.425374    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"]
Oct 31 00:22:09.425608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.425544    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.498494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.498540    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.498626    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmff\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.498690    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.498898    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.498958    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.499039    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.499083    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.499609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.499157    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.558130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.546916    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"]
Oct 31 00:22:09.627967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.627002    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"]
Oct 31 00:22:09.627967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.627767    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.627967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.627856    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.627967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.627968    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.628335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.628022    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.628335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.628109    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.628335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.628202    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.628335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.628244    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.628576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.628356    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmff\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.628576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.628425    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.630145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.629192    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.630145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.629512    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.636003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.634848    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.637293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.636597    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.637293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.637009    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.637293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.637241    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.647410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.647367    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.657930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.657903    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.667681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.665972    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"]
Oct 31 00:22:09.667681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.666121    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.718821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.718608    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmff\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.752525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.752446    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861200    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861306    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861372    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861452    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861569    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861738    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861829    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.861926    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.867455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.862020    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.963952    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964048    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964169    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964208    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964307    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964362    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964405    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964496    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964528    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964601    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964613    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964809 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964784    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.964977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.964954    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.965615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.965593    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.967383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.967361    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.967590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.967568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.978947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.978921    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:09.980112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:09.980084    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:10.141557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:10.139999    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:10.232631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:10.232591    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk" event={"ID":"c7e5fd92-5df1-44cc-a210-1e1534d35281","Type":"ContainerStarted","Data":"3b98b60b11e66e61c78b276acc3e77634bf529182a00d8e6011746b95d9f7d8e"}
Oct 31 00:22:10.308290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:10.308254    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"]
Oct 31 00:22:10.515536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:10.511540    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:10.533502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:10.525680    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:10.533502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:10.525780    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:11.228063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.226921    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" event={"ID":"0038939b-91e8-4b4c-848f-2da60082ff8e","Type":"ContainerStarted","Data":"f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1"}
Oct 31 00:22:11.228063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.226986    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" event={"ID":"0038939b-91e8-4b4c-848f-2da60082ff8e","Type":"ContainerStarted","Data":"e943288e5db6ff3cf1acbe130c0c3e021a72ffea51b64c875577449fb9c58a3f"}
Oct 31 00:22:11.230093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.230062    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk" event={"ID":"c7e5fd92-5df1-44cc-a210-1e1534d35281","Type":"ContainerStarted","Data":"a884750285dae6d1062db66c4c66288268db1d3cd53e3545ea471190d0ee7b8a"}
Oct 31 00:22:11.249443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.249415    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"]
Oct 31 00:22:11.328068 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.328035    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:11.328280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.328111    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:11.328280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.328142    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:11.357156 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:11.318321    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690002f6_4096_42d6_bcf2_250b98c55207.slice/crio-189e77b44d096c9d5c8f9e4d0980735da2cbc11c2ca8ed0968455dfcb8a66a80 WatchSource:0}: Error finding container 189e77b44d096c9d5c8f9e4d0980735da2cbc11c2ca8ed0968455dfcb8a66a80: Status 404 returned error can't find the container with id 189e77b44d096c9d5c8f9e4d0980735da2cbc11c2ca8ed0968455dfcb8a66a80
Oct 31 00:22:11.395272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.395218    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podStartSLOduration=2.395204154 podStartE2EDuration="2.395204154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:11.394776231 +0000 UTC m=+1699.341146948" watchObservedRunningTime="2026-10-31 00:22:11.395204154 +0000 UTC m=+1699.341574870"
Oct 31 00:22:11.459320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.459023    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk" podStartSLOduration=3.4590042260000002 podStartE2EDuration="3.459004226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:11.453749573 +0000 UTC m=+1699.400120316" watchObservedRunningTime="2026-10-31 00:22:11.459004226 +0000 UTC m=+1699.405374948"
Oct 31 00:22:11.546817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.546785    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"]
Oct 31 00:22:11.566514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.566133    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.578560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.577896    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"]
Oct 31 00:22:11.627968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.618827    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-14e39b-openshift-default-dockercfg-rtll2\""
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653481    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653606    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653731    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653781    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653820    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653911    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.653984    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8n74\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.654005    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.654255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.654033    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.754521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.754076    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.755935    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756052    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8n74\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756081    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756106    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756150    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756254    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756400    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756513    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756543    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.756868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.756605    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.757545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.757240    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.757545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.757452    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:11.757545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.757512    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:11.757729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.757548    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.757729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.757663    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.757838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.757785    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.779300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.779246    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.789223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.787205    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.789223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.787831    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8n74\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.798554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.798494    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:11.969211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:11.969178    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:12.459652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.459606    5078 generic.go:356] "Generic (PLEG): container finished" podID="c7e5fd92-5df1-44cc-a210-1e1534d35281" containerID="a884750285dae6d1062db66c4c66288268db1d3cd53e3545ea471190d0ee7b8a" exitCode=0
Oct 31 00:22:12.660582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.639006    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk" event={"ID":"c7e5fd92-5df1-44cc-a210-1e1534d35281","Type":"ContainerDied","Data":"a884750285dae6d1062db66c4c66288268db1d3cd53e3545ea471190d0ee7b8a"}
Oct 31 00:22:12.660582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.639109    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" event={"ID":"690002f6-4096-42d6-bcf2-250b98c55207","Type":"ContainerStarted","Data":"60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8"}
Oct 31 00:22:12.660582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.639146    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" event={"ID":"690002f6-4096-42d6-bcf2-250b98c55207","Type":"ContainerStarted","Data":"189e77b44d096c9d5c8f9e4d0980735da2cbc11c2ca8ed0968455dfcb8a66a80"}
Oct 31 00:22:12.757414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.757373    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:12.758009 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.757438    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:12.769010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.768956    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podStartSLOduration=3.768935002 podStartE2EDuration="3.768935002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:12.742185599 +0000 UTC m=+1700.688556317" watchObservedRunningTime="2026-10-31 00:22:12.768935002 +0000 UTC m=+1700.715305719"
Oct 31 00:22:12.853326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.768992    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:12.853326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.792671    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:12.853326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.792752    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:22:12.853326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:12.769164    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"]
Oct 31 00:22:13.152236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.145045    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:22:13.152236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.151604    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:22:13.152236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.151665    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 31 00:22:13.546798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.546758    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" event={"ID":"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62","Type":"ContainerStarted","Data":"b79386a1f1e770dd7589607df087967c455bde6e0434e7843c32608438f1d0e4"}
Oct 31 00:22:13.729522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.711215    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:13.729522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.711285    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:13.746318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.746062    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:13.746318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.746117    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:13.815718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.753540    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:13.815718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:13.753579    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:14.143417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:14.143012    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:22:14.143417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:14.143064    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 31 00:22:14.605309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:14.605223    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" event={"ID":"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62","Type":"ContainerStarted","Data":"1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5"}
Oct 31 00:22:14.823646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:14.660700    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podStartSLOduration=3.660682276 podStartE2EDuration="3.660682276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:14.660240393 +0000 UTC m=+1702.606611109" watchObservedRunningTime="2026-10-31 00:22:14.660682276 +0000 UTC m=+1702.607052992"
Oct 31 00:22:14.823646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:14.753964    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:14.823646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:14.754010    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:15.553653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.553600    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:15.621489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.608490    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:22:15.621489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.608564    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 31 00:22:15.753682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.753646    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:15.819661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.753697    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:16.083678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.993307    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:22:16.083678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:15.993383    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:27.680411    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:27.928649    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.16:8443/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:27.928731    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/healthz\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.475074    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.475590    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.478451    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.478613    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.478789    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.478813    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.478942    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.478964    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.485150    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.487343    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.490510    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.490555    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.490690    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.490730    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.490912    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.490974    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.491093    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.491149    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.491609    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.495800    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.495845    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.496622    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.496663    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.497992    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.500812    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.394263676s: [/var/lib/containers/storage/overlay/2b42ee51866f81e320b08bf54f164aea3f78f43910e25765893b5c2f47da8c13/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-657fcc8b8f-z9psz_3c0d9ceb-7c2b-4872-a065-4bfe4b332064/manager/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.503757    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.506647    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.506999    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.65:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.512837    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.512878    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.521122    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" podUID="fe713694-148e-4e08-bc8a-becc4f002df5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.55:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.525406    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.525696    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.526031    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.526080    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.526911    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.528298    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.528593    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.528644    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.528698    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podUID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.531638    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.531676    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.533256    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.533294    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:28.533936    5078 logging.go:55] [core] [Channel #1409 SubChannel #1414]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.164:50052", ServerName: "10.132.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.164:50052: operation was canceled"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:28.533983    5078 logging.go:55] [core] [Channel #1412 SubChannel #1417]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.55:50052", ServerName: "10.132.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.55:50052: operation was canceled"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:28.534034    5078 logging.go:55] [core] [Channel #1410 SubChannel #1415]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.65:50052", ServerName: "10.132.0.65:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.65:50052: operation was canceled"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.534382    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.534407    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.986193    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podUID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.986971    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.987027    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.987376    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.987427    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.987607    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.987656    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.988648    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.988683    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.988990    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.61:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.989597    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.989786    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:22:28.179512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:28.990338    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.443066    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.166:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.443200    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.443237    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.443265    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.443290    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.464782    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.163:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.464900    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.464944    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.465832    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.465890    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.475381    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.475711    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.477141    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.477183    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.477412    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.477434    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.481566    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" podUID="6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.56:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.483217    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.483259    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.483542    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.483575    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:29.484657    5078 logging.go:55] [core] [Channel #1418 SubChannel #1419]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.166:50052", ServerName: "10.132.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:59456->10.132.0.166:50052: use of closed network connection"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.485756    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.485895    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.488222    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.488235    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.492700    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.494359    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.494392    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.498270    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.504761    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:29.507645    5078 logging.go:55] [core] [Channel #1404 SubChannel #1405]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47244->10.132.0.160:50052: use of closed network connection"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.509621    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.514563    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.514591    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.525551    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.525817    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.526267    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.526317    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:29.527615    5078 logging.go:55] [core] [Channel #1406 SubChannel #1420]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.67:50052", ServerName: "10.132.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:42590->10.132.0.67:50052: use of closed network connection"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.528623    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.528663    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.568586    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.969816    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.969879    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.988144    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.988199    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.989208    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.989271    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.990294    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.990294    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.990480    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.990536    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.990838    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:29.990871    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.143851    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.143930    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.144285    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.144328    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.179747    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.279567    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.279638    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.443670    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.166:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.454096    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.466853    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.475889    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.475908    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:30.484659    5078 logging.go:55] [core] [Channel #1425 SubChannel #1426]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.166:50052", ServerName: "10.132.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:59470->10.132.0.166:50052: use of closed network connection"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.486688    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.489163    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.493693    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.499143    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.505688    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:30.507607    5078 logging.go:55] [core] [Channel #1430 SubChannel #1431]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47246->10.132.0.160:50052: use of closed network connection"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.523579    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" podUID="6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.56:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.525752    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.525902    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.753012    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.753921    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.754272    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.764760    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.764829    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.767110    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.20:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.767151    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.767450    5078 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.767542    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" podUID="c108abe8-dc34-426e-9a4e-becaf4e882b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.773216    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.773202    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.238066301s: [/var/lib/containers/storage/overlay/91ca6b278e0d098f9fdb6cc0fab978faed6601b8b7e4e51cf7412865093225ee/diff /var/log/pods/openshift-ingress_e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl_9d43026d-495b-4d5c-b30c-4ee2f68bb8f4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.773253    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.773562    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.774304    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.63:50052\" within 1s: context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:22:30.774431    5078 logging.go:55] [core] [Channel #1433 SubChannel #1434]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.63:50052", ServerName: "10.132.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.63:50052: operation was canceled"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.784338    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.249160204s: [/var/lib/containers/storage/overlay/2dd6c053aaef3892e80fe73249f3b99f23cc1323b3509309cbbd67a1cb1eb6ae/diff /var/log/pods/openshift-monitoring_node-exporter-kpj57_c25f317f-5d84-42c6-b06d-e69dbb04420e/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.839720    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.840413    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.840766    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.16:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.840809    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.863520    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.863578    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.969717    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.969768    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:30.990767    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.476451    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.487581    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.526094    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.526111    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.754156    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.754157    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.755236    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.773814    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.784072    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.784129    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.840680    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.961906    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.143583    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.143643    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.201163    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.325782    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.376315    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.376292    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.376375    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:52.845038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:32.376621    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" podUID="fe713694-148e-4e08-bc8a-becc4f002df5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.55:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:58.215099    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:58.215174    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.449210    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-s8g6n" podUID="65fab40a-d3b0-4bf1-9ab3-396dfccebe5f" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.762827    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.763169    5078 patch_prober.go:29] interesting pod/image-registry-6c895f9d65-tkw4d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.763209    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" podUID="2f6a88c2-1453-4033-8027-afc8b69d3ee7" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.763437    5078 patch_prober.go:29] interesting pod/image-registry-6c895f9d65-tkw4d container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.763485    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" podUID="2f6a88c2-1453-4033-8027-afc8b69d3ee7" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.763717    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.763947    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.764158    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.770984    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": write tcp 10.132.0.2:48334->10.132.0.68:8443: write: broken pipe"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.771147    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qnjc2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.771174    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.152:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.804310    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.804757    5078 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.804793    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" podUID="c108abe8-dc34-426e-9a4e-becaf4e882b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.804961    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": EOF"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.820390    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.820425    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.826641    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.826674    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:31.771736    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.861353    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:22:59.861385    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.133224    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.133902    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.133947    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.134933    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.134975    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.136270    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.141515    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.925582341s: [/var/lib/containers/storage/overlay/78f9faa2af0fbe533a7c915e2137bb51c7dcab109d61a78aa0a24662b7b019b6/diff ]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.148777    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.927740654s: [/var/lib/containers/storage/overlay/a9306ec51048b5e52797249e26854515341da20b8d30a5b224cfeb71b45488a3/diff ]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.148853    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.148889    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.149501    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.149538    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.151844    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.153562    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.153598    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.153787    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.155550    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" podUID="6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.56:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.155824    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.155896    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.156138    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.157207    5078 logging.go:55] [core] [Channel #1408 SubChannel #1413]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.61:50052", ServerName: "10.132.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:56946->10.132.0.61:50052: use of closed network connection"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.164857    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.164885    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.20:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.164918    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.165263    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.171998    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.172029    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.172202    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.172223    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.172562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.174371    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.174664    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.175830    5078 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.175885    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b" podUID="922f74f1-f495-4764-b2c9-1f40b2bcf849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.178633    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.181392    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.181438    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.182806    5078 patch_prober.go:29] interesting pod/network-check-target-2qs9j container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.132.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.182839    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-2qs9j" podUID="362b6ff8-9785-49c5-8704-db4c94a603d9" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.132.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.183034    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.183271    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.62:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.183344    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.183650    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.61:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.184113    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.184135    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.187360    5078 patch_prober.go:29] interesting pod/dns-default-shskm container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.187383    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-shskm" podUID="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.188694    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.189454    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.065130612s: [/var/lib/containers/storage/overlay/4c51c9f57f581a6e75b6d3a094566e0f6463f80d968868006cc52e8ee886f718/diff /var/log/pods/e2e-models-e2e-worker-w6-0402df_distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd_6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.189616    5078 logging.go:55] [core] [Channel #1449 SubChannel #1453]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.164:50052", ServerName: "10.132.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.164:50052: operation was canceled"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.190166    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.190306    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.190526    5078 logging.go:55] [core] [Channel #1428 SubChannel #1429]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.55:50052", ServerName: "10.132.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:55952->10.132.0.55:50052: use of closed network connection"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.190581    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.190606    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.190656    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.191405    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.192511    5078 logging.go:55] [core] [Channel #1403 SubChannel #1407]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.163:50052", ServerName: "10.132.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:34656->10.132.0.163:50052: use of closed network connection"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.193290    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podUID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": dial tcp 10.132.0.34:8081: connect: connection refused"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.193545    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.194101    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.66:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.194420    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.211330    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.196731    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.63:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.195399    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeNotReady"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.211506    5078 setters.go:546] "Node became not ready" node="ip-10-0-143-16.us-east-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T00:23:00Z","lastTransitionTime":"2026-10-31T00:23:00Z","reason":"KubeletNotReady","message":"container runtime is down"}
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.194943    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.214155    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.205849    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 31.670841924s: [/var/lib/containers/storage/overlay/28367acf4a29f13cef49e68bc7493dea1509daf885d2e0345a1f29d4c3652438/diff /var/log/pods/openshift-console_downloads-6dd789c8b7-6wdkk_b6209fdf-df1c-4a4f-b5ce-55812e9f07a4/download-server/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.207185    5078 logging.go:55] [core] [Channel #1442 SubChannel #1457]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.66:50052", ServerName: "10.132.0.66:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.66:50052: operation was canceled"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.207572    5078 logging.go:55] [core] [Channel #1437 SubChannel #1459]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.63:50052", ServerName: "10.132.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.63:50052: operation was canceled"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.199976    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podUID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": dial tcp 10.132.0.34:8081: connect: connection refused"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.215012    5078 logging.go:55] [core] [Channel #1436 SubChannel #1444]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.61:50052", ServerName: "10.132.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.61:50052: operation was canceled"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.224420    5078 logging.go:55] [core] [Channel #1435 SubChannel #1443]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.62:50052", ServerName: "10.132.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.62:50052: operation was canceled"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.872603    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.872667    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.872712    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.872822    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.65:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.876582    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.879246    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.16:8443/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.879282    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/healthz\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.879520    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.880653    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.880790    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.880850    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.880881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.880992    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.881022    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.881429    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.881478    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.881781    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.881821    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.886843    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.887174    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.887555    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.887625    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.887676    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.887939    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.888909    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 32.353929709s: [/var/lib/containers/storage/overlay/8677fc8bbce162a6aefbe38a5ff7136e8878fb461993fcb155eb8e3457dce3c9/diff /var/log/pods/e2e-models-e2e-worker-w1-4c360e_premium-w1-4c360e-kserve-74fd457656-4pqtd_75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.889207    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.889254    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.889408    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.889436    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.889654    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.694935772s: [/var/lib/containers/storage/overlay/cd98f1d227f81f9918bbe843c431beecebf10095b1ba4e29a59a31f2b0c73e43/diff /var/log/pods/openshift-ingress_e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5_cf087db0-eaae-4c32-8e65-4a6238b2e282/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890221    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.645094222s: [/var/lib/containers/storage/overlay/73885c2717c72d927646139d68a37f1abe006f19fe5d489446bd05e7ec999241/diff /var/log/pods/openshift-ingress_e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z_4b20778e-6adf-46f7-ae50-b389aa95f36c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890251    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 32.355106961s: [/var/lib/containers/storage/overlay/0227579f57fd06bb371e4cac90a626c073e3b092804f5b41ad5095850836fb73/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-5x24b_0c1437fb-5c91-492e-ae01-54983042358a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890278    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.099161494s: [/var/lib/containers/storage/overlay/cbe4c355fa3b54095a696dac9c2acfe4fd917105bf3d7dc6ee2b0f61c2df24fb/diff /var/log/pods/e2e-models-e2e-worker-w1-4c360e_distinct-2-w1-4c360e-kserve-759c984544-2stvb_869264e1-101e-4adb-b8f6-17bfa992b217/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890299    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037639837s: [/var/lib/containers/storage/overlay/266712d39d629ca24e9faf0a87fe0d9b2502a139f9c15ecd33ba3167bf616eb6/diff /var/log/pods/openshift-network-diagnostics_network-check-target-2qs9j_362b6ff8-9785-49c5-8704-db4c94a603d9/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:04.895653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890325    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.040840619s: [/var/lib/containers/storage/overlay/ab6e76ce2d5d04a9855462f8963e58f6cb401235bb2fcb996f841113a664dbf1/diff /var/log/pods/openshift-ingress_e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp_1f1c1cec-fbce-4694-bb48-b6c32800f1e5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890344    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037683626s: [/var/lib/containers/storage/overlay/b16aaef2dec7ab4301233a0fd2186ceb2e0595b43c8ff285a129e861198f0b37/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hqmv6_1a1a8190-b44f-41ac-8f0f-5020e6e27c18/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890364    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.675561627s: [/var/lib/containers/storage/overlay/5f6071c29b1c38fc9b7e6593cb8740fe139416edb379d5ac215ead9d31f59182/diff /var/log/pods/openshift-ingress_e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j_cd56ff0f-da7a-4de9-b2a0-125e471de6d2/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890390    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037727619s: [/var/lib/containers/storage/overlay/1a98c176f83f46e35891f8acc0557b6e2c50f40a1c1b1fdd8d263d949b2270ce/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-68c79_6b213ff9-3e00-426a-901f-e5ca105d37e8/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890411    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037747682s: [/var/lib/containers/storage/overlay/909551a695c4e91f2197c1eb736ea408c8f2771e7f4ec5283649d2cbce7546a5/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hqmv6_1a1a8190-b44f-41ac-8f0f-5020e6e27c18/nbdb/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890438    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037766424s: [/var/lib/containers/storage/overlay/d24052ff235a855d9503124a3ce931cb4f77f0632d962470feecdb32b0abc75a/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-16.us-east-2.compute.internal_f6d13e136b05ce96356f6b9a00f72419/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890512    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037784298s: [/var/lib/containers/storage/overlay/6104027f232a4850a842843fa5bd1b6f5ed002aae3215dc58896d0b90acff03a/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-7bbnp_dd07a3a8-5fa2-41eb-afcd-4880f4f3cd58/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890539    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037861084s: [/var/lib/containers/storage/overlay/8248e940483064dcb188dd9cc9d2fbffbba3da07e6e511b52cfc2305b43c1a77/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hqmv6_1a1a8190-b44f-41ac-8f0f-5020e6e27c18/northd/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890560    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.037880021s: [/var/lib/containers/storage/overlay/c9e473f3101f25c7ef59abcb4b32cd21f92f7fa8ab1caf7f2605ebd77bcfc398/diff /var/log/pods/opendatahub_maas-controller-7b8bb6f8d-jrv29_68cb91e9-c6e4-460f-9318-49d79080dedb/manager/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890647    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.036033836s: [/var/lib/containers/storage/overlay/b90fe7ba0cfde5e6166cd5972ea93bbb1ead975d2da161f2be2c06ebb1850975/diff /var/log/pods/openshift-multus_network-metrics-daemon-vl6sr_5f6789dc-0c9b-4507-9615-93c3ec632af2/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890693    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.038018931s: [/var/lib/containers/storage/overlay/30742caa280ac316878d7010853251516f108e1bb63822617442be5c222d97d9/diff /var/log/pods/openshift-multus_multus-4f6sj_d1090e22-f04c-4bda-b340-0966209254af/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890867    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.033997941s: [/var/lib/containers/storage/overlay/44747a9da9af7050986affd008fc829377a4625b8c569a1e1d5e338ba610bba2/diff /var/log/pods/kube-system_global-pull-secret-syncer-ktssl_e73a9af6-0664-409b-a096-83a990ec98ce/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.890972    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034103833s: [/var/lib/containers/storage/overlay/c8a35f60807def4a0f568051dd0a959db5617efb0f0a351a9d4b5da82f703773/diff /var/log/pods/kube-system_konnectivity-agent-s8g6n_65fab40a-d3b0-4bf1-9ab3-396dfccebe5f/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.891043    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034172829s: [/var/lib/containers/storage/overlay/f1f2e2c6ebc272c7fad35817f642ad1e071e064fd8bebe1e1bfc244bc0477f08/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hqmv6_1a1a8190-b44f-41ac-8f0f-5020e6e27c18/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.891122    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034059472s: [/var/lib/containers/storage/overlay/2b3fba2a8b5ee9b7d72888e553bbcc0e6081db40505979adf454203317640153/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l5xz9_7e84018f-f7a2-4588-bd5b-b67814c509bd/manager/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.891288    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 45.013245736s: [/var/lib/containers/storage/overlay/611716ea6981e11eb7ac68b92178186f89c02b94d80fa042e7d3ed43b7dc2d00/diff /var/log/pods/opendatahub_ai-gateway-controller-86f7d849b4-vgljn_6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152/manager/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.891323    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.675430098s: [/var/lib/containers/storage/overlay/6e52cf805eed37a4d96f2623ca1acb40227c3b860ebdd3f1ddd5dc78a3689caa/diff /var/log/pods/e2e-models-e2e-worker-w4-d91de6_simulator-w4-d91de6-kserve-5656898d8-pdcdk_66848bbb-e0ec-48b2-9154-55eec47a4122/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.891515    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.120225098s: [/var/lib/containers/storage/overlay/c9232f3d186cba9d1bdb655107ba41d7f817b362f38e9418806f05e8e1af6af8/diff /var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mpl9v_e9bb7055-cf51-4182-a092-2364ed3d5444/manager/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892180    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.056491388s: [/var/lib/containers/storage/overlay/9edbae5efe40eae06cb4f7977884456f01ffd27a39260f6839a24afc25f39044/diff /var/log/pods/openshift-ingress_e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9_5de1af60-a588-4e7e-b8cc-b201871af022/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892220    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.071854964s: [/var/lib/containers/storage/overlay/204c1e60944e5db5e9321b1f3b6a777d025f09f73f97972f0b8699985f9747d4/diff /var/log/pods/e2e-models-e2e-worker-w2-7c5da3_simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk_641786fc-cd71-417f-a8b2-153079f8b512/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892235    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.256864301s: [/var/lib/containers/storage/overlay/e4f963574e8a7be9bd3d0d8191818de219e291bd04808acc1f11c9ff89892316/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-h2j9x_964f2946-7176-4f03-b41f-03bd65aa8346/cluster-samples-operator/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892256    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.633534729s: [/var/lib/containers/storage/overlay/ea0a358f4aee6656c5842f147a8b5270a878142dfbdc250062f6bf1395b07ff2/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-h2j9x_964f2946-7176-4f03-b41f-03bd65aa8346/cluster-samples-operator-watch/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892262    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.678138203s: [/var/lib/containers/storage/overlay/1d63d2a2bb01844ab52cbe115dfd989c9dd97c3643f3ea7e0467a55418555a96/diff /var/log/pods/openshift-dns_dns-default-shskm_d1384f8a-a2ad-4e55-b6e5-14ab134fa489/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892295    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.087571218s: [/var/lib/containers/storage/overlay/3961e25225631d153fe49e1f3a5ba08cf45dfa3a4b22823e944ad6d2c33e7707/diff /var/log/pods/kuadrant-system_authorino-58544c6c77-l9nc7_fdac0d2e-793d-4fad-a635-528229ed9df1/authorino/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892414    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892451    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892528    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892549    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892658    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892678    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.892892    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.085808656s: [/var/lib/containers/storage/overlay/b3d078bccd606a9942c6d9c84c23e619507bd17993f50ba75a8a29ee956ab00d/diff /var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-k7rhq_1b949609-5560-4b48-a5b7-229d7430d7c0/networking-console-plugin/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.893391    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.086323066s: [/var/lib/containers/storage/overlay/2b7c457b7f9c035a53b7e8ad0dd5db1142b7f416c23cb43be8976f8bfb1c847d/diff /var/log/pods/openshift-ingress_e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n_905c8138-3bf0-41f0-b1db-fa377aa449c3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.893698    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.087932387s: [/var/lib/containers/storage/overlay/7ada997914f6dbdccdbbff30b6e3b1d847b7af43fd65f94dc0044ccd4219766e/diff /var/log/pods/openshift-ingress_e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr_48402b00-cb93-46fe-87c6-4e1437aca83c/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.894078    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:00.894318    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1eaf8e_40da_42fe_bb27_6900f2488745.slice/crio-e7e6133ee19f34df4a5dd9f4849410333a6e88347215a342cacd944d8a0ea28d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1eaf8e_40da_42fe_bb27_6900f2488745.slice/crio-conmon-e7e6133ee19f34df4a5dd9f4849410333a6e88347215a342cacd944d8a0ea28d.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.894637    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.679833049s: [/var/lib/containers/storage/overlay/0d51f1c430df26218479f98ec5d6d019c5213c871fdc82acc8826dd1af63ec41/diff /var/log/pods/e2e-models-e2e-worker-w5-7b2341_premium-w5-7b2341-kserve-bb7676bf4-2xzq4_70a9ea32-a1a7-41eb-b335-1203dc85faaa/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.895016    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.068258117s: [/var/lib/containers/storage/overlay/43a5cb4fe0f05b212b500d91385881da88ac4b435add4e8332e1037f1cb4110e/diff /var/log/pods/e2e-models-e2e-worker-w6-0402df_simulator-w6-0402df-kserve-84cd889645-8tqm4_b0dc259b-ccb1-4912-9dc8-c52852f55e75/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.898160    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.898208    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.899590    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.899649    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.902111    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.902676    5078 logging.go:55] [core] [Channel #1478 SubChannel #1479]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.67:50052", ServerName: "10.132.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.67:50052: operation was canceled"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.913631    5078 logging.go:55] [core] [Channel #1456 SubChannel #1458]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.65:50052", ServerName: "10.132.0.65:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:33914->10.132.0.65:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.918588    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:00.922611    5078 logging.go:55] [core] [Channel #1450 SubChannel #1454]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:60138->10.132.0.160:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.928574    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.928603    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.929536    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.929565    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.936573    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.936605    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.939578    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.970170    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.970223    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:00.989132    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.919349668s: [/var/lib/containers/storage/overlay/1d5e1e44082f62a03a6394c66ac05b38b49cd46ad8a7613d8170ca85dee89b39/diff /var/log/pods/openshift-ingress_e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk_86073226-02be-4443-b886-654281e0a89a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.094487    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.094539    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.133920    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.133946    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.137030    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.137652    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.138443    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.166:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.139505    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.139518    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.139548    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.140838    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.744879967s: [/var/lib/containers/storage/overlay/6386d20df6510f4b8326c4b9786d7b92dc7a4d847b8f6f1094b898dae4eedfbd/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p_325f8eed-03c3-4f20-a72a-7753e560cb45/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.150131    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.154266    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.155364    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.156704    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.165306    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.165346    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.172873    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.174384    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.174497    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.175397    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.175423    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.175442    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.163:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.178593    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" podUID="6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.56:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.178748    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.179619    5078 logging.go:55] [core] [Channel #1439 SubChannel #1440]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.166:50052", ServerName: "10.132.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:53222->10.132.0.166:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.180580    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.180621    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.184454    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.61:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.184503    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.62:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.189602    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.189637    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.190595    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.190616    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.190628    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.190667    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.190693    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.192045    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.194566    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.194594    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.196548    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" podUID="6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.56:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.204560    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.205580    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.413207766s: [/var/lib/containers/storage/overlay/5fd2abb766f9cdb8e86b57bd64236f9ee6fcf9055ede978283ff6a848c3c001b/diff /var/log/pods/openshift-ingress_e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq_bba615fc-186e-48cb-9bf1-18ee9c33dc9a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.205594    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.20:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.205606    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.205628    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.207990    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.210175    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.66:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.211682    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.63:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.215624    5078 logging.go:55] [core] [Channel #1445 SubChannel #1447]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:33986->10.132.0.64:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.217618    5078 logging.go:55] [core] [Channel #1446 SubChannel #1448]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.163:50052", ServerName: "10.132.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:43180->10.132.0.163:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.223604    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.223652    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.226240    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.226602    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.226618    5078 logging.go:55] [core] [Channel #1451 SubChannel #1461]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.62:50052", ServerName: "10.132.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51306->10.132.0.62:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.226629    5078 logging.go:55] [core] [Channel #1452 SubChannel #1462]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.61:50052", ServerName: "10.132.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:35674->10.132.0.61:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.226634    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.228873    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" podUID="fe713694-148e-4e08-bc8a-becc4f002df5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.55:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.250632    5078 logging.go:55] [core] [Channel #1465 SubChannel #1470]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.66:50052", ServerName: "10.132.0.66:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:39484->10.132.0.66:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.252620    5078 logging.go:55] [core] [Channel #1468 SubChannel #1469]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.63:50052", ServerName: "10.132.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:56940->10.132.0.63:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:01.269626    5078 logging.go:55] [core] [Channel #1475 SubChannel #1476]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.55:50052", ServerName: "10.132.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52810->10.132.0.55:50052: use of closed network connection"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.345585    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" podUID="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" containerName="nbdb" probeResult="failure" output="command timed out"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.345730    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hqmv6" podUID="1a1a8190-b44f-41ac-8f0f-5020e6e27c18" containerName="sbdb" probeResult="failure" output="command timed out"
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.458577    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.878886405s: [/var/lib/containers/storage/overlay/1305a930c0a4cfc5e5a11a0cf862a794c68b97f9471bf73b03990dde5ab161aa/diff /var/log/pods/openshift-ingress_e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz_22f1a149-d6c7-4c5f-97c1-639866c3b6f5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.707537    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 30.379132827s: [/var/lib/containers/storage/overlay/d440d6319484ec2175cf5dc4823427ac14a5d5794afde442c03cd3f121c35b05/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6_a75e181d-6850-4b90-a34a-3a1d098a6f53/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.760928    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.968397905s: [/var/lib/containers/storage/overlay/9877baa4d2140b3d58bb4d38fceb5148bd60434c27c68a0f877301b9d61a9625/diff /var/log/pods/openshift-ingress_e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw_85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.810303    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 33.275294137s: [/var/lib/containers/storage/overlay/72d1211b3cfb48a8d28ec5a8bf1520b60167ae8d6570f1c50043f403c665f65a/diff /var/log/pods/openshift-ingress_e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh_86417f75-420a-4ef6-84ee-1d36bd8632ca/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.969818    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:07.924549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.969881    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.970583    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:01.970625    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.062977    5078 trace.go:236] Trace[2130111898]: "Calculate volume metrics of tmp for pod openshift-console/downloads-6dd789c8b7-6wdkk" (31-Oct-2026 00:23:00.221) (total time: 1841ms):
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: Trace[2130111898]: [1.841421991s] [1.841421991s] END
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.184745    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.62:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:02.225613    5078 logging.go:55] [core] [Channel #1480 SubChannel #1481]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.62:50052", ServerName: "10.132.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51328->10.132.0.62:50052: use of closed network connection"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.245587    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.463015    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podUID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": dial tcp 10.132.0.34:8081: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.771656    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.823781    5078 trace.go:236] Trace[38973815]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-6c895f9d65-tkw4d" (31-Oct-2026 00:23:00.218) (total time: 2605ms):
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: Trace[38973815]: [2.605146977s] [2.605146977s] END
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.970313    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:02.970369    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.175558    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.175610    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.191452    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.213564    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.213622    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.229589    5078 patch_prober.go:29] interesting pod/dns-default-shskm container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.229644    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-shskm" podUID="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.759933    5078 trace.go:236] Trace[1536818867]: "iptables ChainExists" (31-Oct-2026 00:23:00.184) (total time: 3575ms):
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: Trace[1536818867]: [3.575903505s] [3.575903505s] END
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.759934    5078 trace.go:236] Trace[541155830]: "iptables ChainExists" (31-Oct-2026 00:23:00.170) (total time: 3589ms):
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: Trace[541155830]: [3.589363314s] [3.589363314s] END
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.970401    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:03.970452    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:04.806017    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:04.846645    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:04.969852    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:04.969894    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.133005    5078 scope.go:122] "RemoveContainer" containerID="fdd1f30822e3381ed10c187ada98072a3c6c51ae970476f982da2a5969064b05"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.152787    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.213992    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-s8g6n" podUID="65fab40a-d3b0-4bf1-9ab3-396dfccebe5f" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.237669    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-s8g6n" podUID="65fab40a-d3b0-4bf1-9ab3-396dfccebe5f" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.239172    5078 patch_prober.go:29] interesting pod/dns-default-shskm container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.132.0.17:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.239218    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-shskm" podUID="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.17:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.239680    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:05.742134    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="38.062s"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.742224    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.743858    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.761296    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.761358    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d79769d5-nfjkq"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.761424    5078 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.825895    5078 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="prometheus-operator-admission-webhook" containerStatusID={"Type":"cri-o","ID":"b74c99cff5553b203610ed6531ef7db49fc1ff64817a62ae04ea5be20bd7ce64"} pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" containerMessage="Container prometheus-operator-admission-webhook failed liveness probe, will be restarted"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.825991    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" containerID="cri-o://b74c99cff5553b203610ed6531ef7db49fc1ff64817a62ae04ea5be20bd7ce64" gracePeriod=30
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.828642    5078 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"a6e89c3847eef8d6df2a2d3fd02f92c307a2828a4ca73c740c118e90d1ea7f0d"} pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" containerMessage="Container console-operator failed liveness probe, will be restarted"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.828695    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" containerID="cri-o://a6e89c3847eef8d6df2a2d3fd02f92c307a2828a4ca73c740c118e90d1ea7f0d" gracePeriod=30
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.972070    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:05.972126    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.167663    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5e64c1-0326-4a0b-97c6-87ff734a6521" path="/var/lib/kubelet/pods/6c5e64c1-0326-4a0b-97c6-87ff734a6521/volumes"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.220638    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.220691    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.254868    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.254917    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.271632    5078 patch_prober.go:29] interesting pod/dns-default-shskm container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.271681    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-shskm" podUID="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.491514    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.752703    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.752748    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.970453    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:06.970551    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.041357    5078 generic.go:356] "Generic (PLEG): container finished" podID="6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152" containerID="775f4e59b730affd598dac624b3243c1b80a7ae6a20e2796c6aec75089940bec" exitCode=1
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.043302    5078 generic.go:356] "Generic (PLEG): container finished" podID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerID="b74c99cff5553b203610ed6531ef7db49fc1ff64817a62ae04ea5be20bd7ce64" exitCode=0
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.045059    5078 generic.go:356] "Generic (PLEG): container finished" podID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerID="89597f02c6c654dd72b20e98d8d535d4d2311ad7d44e1b6083f0a6fead8c7eb2" exitCode=1
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.047105    5078 generic.go:356] "Generic (PLEG): container finished" podID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerID="e7e6133ee19f34df4a5dd9f4849410333a6e88347215a342cacd944d8a0ea28d" exitCode=1
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.228937    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.228970    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" event={"ID":"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152","Type":"ContainerDied","Data":"775f4e59b730affd598dac624b3243c1b80a7ae6a20e2796c6aec75089940bec"}
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.229042    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.229156    5078 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.229172    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.229393    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.229451    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.230396    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.230450    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" event={"ID":"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0","Type":"ContainerDied","Data":"b74c99cff5553b203610ed6531ef7db49fc1ff64817a62ae04ea5be20bd7ce64"}
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.230501    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.230674    5078 scope.go:122] "RemoveContainer" containerID="775f4e59b730affd598dac624b3243c1b80a7ae6a20e2796c6aec75089940bec"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.231537    5078 scope.go:122] "RemoveContainer" containerID="89597f02c6c654dd72b20e98d8d535d4d2311ad7d44e1b6083f0a6fead8c7eb2"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.240661    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.240697    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.240772    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.241643    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.241668    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.242333    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.242377    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfnp\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.242414    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.242433    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358573    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358635    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358731    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358772    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfnp\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358837    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358912    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.358963    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.359002    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:07.507966    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.572875    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.573022    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667107    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667269    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667420    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667513    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667653    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsp7\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667815    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.667978    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.668032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.668094    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770280    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsp7\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770395    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770519    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770561    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770601    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770717    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770764    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770825    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.770846    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:07.831079    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930553    5078 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930599    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930622    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930638    5078 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930660    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" event={"ID":"3c0d9ceb-7c2b-4872-a065-4bfe4b332064","Type":"ContainerDied","Data":"89597f02c6c654dd72b20e98d8d535d4d2311ad7d44e1b6083f0a6fead8c7eb2"}
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930729    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930864    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930903    5078 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930931    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" event={"ID":"bf1eaf8e-40da-42fe-bb27-6900f2488745","Type":"ContainerDied","Data":"e7e6133ee19f34df4a5dd9f4849410333a6e88347215a342cacd944d8a0ea28d"}
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930947    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930966    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e358a6-9hnmp"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.930997    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d98f6f1-xglx8"]
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.932601    5078 scope.go:122] "RemoveContainer" containerID="e7e6133ee19f34df4a5dd9f4849410333a6e88347215a342cacd944d8a0ea28d"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.936757    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-719e12-openshift-default-dockercfg-vlkg9\""
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.951274    5078 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="router" containerStatusID={"Type":"cri-o","ID":"6a8f117cb1730c3a9d8dd12471c1758fe1e91086c7d36273454a6d9e1a23fd4b"} pod="openshift-ingress/router-default-74cc75bf99-fkb87" containerMessage="Container router failed liveness probe, will be restarted"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.970533    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.970574    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.975678    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.975828    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.975877    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.975965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.976059    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.976076    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.976323    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.976378    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpm7\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.976432    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:07.999523    5078 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="download-server" containerStatusID={"Type":"cri-o","ID":"f210307b6ed0f7d4f6c22410102133a604ac8a14823f1c02613cdcb331ca370d"} pod="openshift-console/downloads-6dd789c8b7-6wdkk" containerMessage="Container download-server failed liveness probe, will be restarted"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.079605    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.079697    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpm7\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.079774    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.079848    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.080041    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.080095    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.080239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.379668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.080437    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.080482    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.223535    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" containerID="cri-o://6a8f117cb1730c3a9d8dd12471c1758fe1e91086c7d36273454a6d9e1a23fd4b" gracePeriod=10
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.223553    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" containerID="cri-o://f210307b6ed0f7d4f6c22410102133a604ac8a14823f1c02613cdcb331ca370d" gracePeriod=5
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.256843    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.256835    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ce096e-fcb0-471b-a842-382de2ba1100" path="/var/lib/kubelet/pods/e2ce096e-fcb0-471b-a842-382de2ba1100/volumes"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.480115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.574712    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.574891    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.638298    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2dab808-935c-4922-a6a7-407540897107" path="/var/lib/kubelet/pods/e2dab808-935c-4922-a6a7-407540897107/volumes"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.645035    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.652621    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.653183    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.661331    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.666456    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.666517    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"]
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.666669    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"]
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.666752    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"]
Oct 31 00:23:08.668259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.666903    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"]
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.690042    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfnp\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.690337    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.690355    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.693767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.693819    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.693944    5078 scope.go:122] "RemoveContainer" containerID="e9348376250757d9b8aac40faa34b41ea15b689900b61bbe51954124e74596b0"
Oct 31 00:23:08.695961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.694160    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsp7\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.724765    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.724934    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.724953    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729783    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729806    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729836    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729855    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729875    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50d42ee-bwjc8"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729948    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.729988    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.730022    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1f9f39-crndx"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.730084    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.730184    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.730241    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.730271    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3041f7fd-k66fj"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.731776    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.731813    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b4292c-zg9rh"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.731907    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.731952    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.732002    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.732025    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.732174    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.732632    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.733108    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.734003    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.736046    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.736849    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.736890    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741749    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741776    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741809    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741858    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741882    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741921    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741938    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741962    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.741985    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742030    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742069    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742090    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742106    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742131    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742150    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742167    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742198    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742222    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742259    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742296    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742373    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742397    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742423    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742491    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742509    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742529    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742555    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e784d715-njbwf"]
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742576    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742592    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742623    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742666    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742693    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742717    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742764    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742789    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742814    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s8g6n"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742835    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742883    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742931    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-shskm"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.742981    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.743026    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.743640    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.749242    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.749294    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.749340    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.749381    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.749421    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.749477    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s8g6n"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.752680    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-945abd-openshift-default-dockercfg-fzmhz\""
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.757436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.771754    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.771977    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-shskm"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.772942    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.773066    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.773346    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.773485    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.773916    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:23:08.777719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.773974    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.782733    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.787596    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.787602    5078 scope.go:122] "RemoveContainer" containerID="93fd29d880abd6e675c1f72cb4419f8265314c7ae689a18ef2aa22df2413f7db"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.790875    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.805612    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.826105    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.836264    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.839054    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpm7\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.839289    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-vj72g\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.840377    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.857085    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.857627    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.860626    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:23:08.882080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.868843    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:23:08.946915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.945217    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.956084    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.956207    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcwm\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.957073    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.957152    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.957273    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.957416    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.960775    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.960838    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.963481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.961085    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:08.971344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.970750    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:08.971344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.970812    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:08.978383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:08.978278    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.080775    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt\" (UniqueName: \"kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt\") pod \"c7e5fd92-5df1-44cc-a210-1e1534d35281\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") "
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.080888    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca\") pod \"c7e5fd92-5df1-44cc-a210-1e1534d35281\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") "
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081021    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca\") pod \"c7e5fd92-5df1-44cc-a210-1e1534d35281\" (UID: \"c7e5fd92-5df1-44cc-a210-1e1534d35281\") "
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081711    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081814    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081878    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081922    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.081950    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.082023    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.082282    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.085792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.082358    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcwm\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.087063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.087040    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:09.124978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.124943    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk" event={"ID":"c7e5fd92-5df1-44cc-a210-1e1534d35281","Type":"ContainerDied","Data":"3b98b60b11e66e61c78b276acc3e77634bf529182a00d8e6011746b95d9f7d8e"}
Oct 31 00:23:09.124978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.124961    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"
Oct 31 00:23:09.125218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.125031    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b98b60b11e66e61c78b276acc3e77634bf529182a00d8e6011746b95d9f7d8e"
Oct 31 00:23:09.128548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.128456    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca" pod "c7e5fd92-5df1-44cc-a210-1e1534d35281" (UID: "c7e5fd92-5df1-44cc-a210-1e1534d35281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:23:09.184349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.182914    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.185126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.185102    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:09.185328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.185303    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.185436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.185337    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.185714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.185695    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.215500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.215337    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.215500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.215365    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.216011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.215987    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.216397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.216370    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcwm\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.216869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.216840    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo\") pod \"e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") " pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.268936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.268904    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt" pod "c7e5fd92-5df1-44cc-a210-1e1534d35281" (UID: "c7e5fd92-5df1-44cc-a210-1e1534d35281"). InnerVolumeSpecName "kube-api-access-r9kbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:09.269194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.269173    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca" pod "c7e5fd92-5df1-44cc-a210-1e1534d35281" (UID: "c7e5fd92-5df1-44cc-a210-1e1534d35281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:23:09.286980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.286947    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e5fd92-5df1-44cc-a210-1e1534d35281-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:09.286980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.286976    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kbt\" (UniqueName: \"kubernetes.io/projected/c7e5fd92-5df1-44cc-a210-1e1534d35281-kube-api-access-r9kbt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:09.398070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.359661    5078 scope.go:122] "RemoveContainer" containerID="f1375076c0cef7628890747b1ec1fa485d8748955209cf368d0a47838fb08453"
Oct 31 00:23:09.492928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.492804    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:09.493712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.493672    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:23:09.538804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.538155    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/1.log"
Oct 31 00:23:09.538804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.538210    5078 generic.go:356] "Generic (PLEG): container finished" podID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerID="a6e89c3847eef8d6df2a2d3fd02f92c307a2828a4ca73c740c118e90d1ea7f0d" exitCode=1
Oct 31 00:23:09.538804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.538556    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" event={"ID":"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb","Type":"ContainerDied","Data":"a6e89c3847eef8d6df2a2d3fd02f92c307a2828a4ca73c740c118e90d1ea7f0d"}
Oct 31 00:23:09.538804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.538623    5078 scope.go:122] "RemoveContainer" containerID="887f478b217938bcd372eee50aca8ef1d9aa1fb32d391ef82d7c0b4cc6d58160"
Oct 31 00:23:09.542743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.542368    5078 generic.go:356] "Generic (PLEG): container finished" podID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerID="f210307b6ed0f7d4f6c22410102133a604ac8a14823f1c02613cdcb331ca370d" exitCode=0
Oct 31 00:23:09.547451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.545520    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6wdkk" event={"ID":"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4","Type":"ContainerDied","Data":"f210307b6ed0f7d4f6c22410102133a604ac8a14823f1c02613cdcb331ca370d"}
Oct 31 00:23:09.845622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.845584    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:09.845819 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.845637    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:09.970391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.970352    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:09.970923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:09.970888    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:10.220117 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.220067    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026df7e1-ad6c-4ee6-bf8e-26585908b64c" path="/var/lib/kubelet/pods/026df7e1-ad6c-4ee6-bf8e-26585908b64c/volumes"
Oct 31 00:23:10.250865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.250838    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-east-2.compute.internal" event="NodeReady"
Oct 31 00:23:10.403755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.403621    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf6bac6-e10a-46a4-9a49-d97dfff92d51" path="/var/lib/kubelet/pods/6bf6bac6-e10a-46a4-9a49-d97dfff92d51/volumes"
Oct 31 00:23:10.502744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.502716    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6cfda0-f8f3-44f2-b858-badbe40119c6" path="/var/lib/kubelet/pods/6e6cfda0-f8f3-44f2-b858-badbe40119c6/volumes"
Oct 31 00:23:10.565048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.565008    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused" start-of-body=
Oct 31 00:23:10.565220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.565071    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused"
Oct 31 00:23:10.739173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.739144    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc912476-3494-48fa-be70-0156e78663b9" path="/var/lib/kubelet/pods/bc912476-3494-48fa-be70-0156e78663b9/volumes"
Oct 31 00:23:10.793646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.793608    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d084ee93-37d1-4c1b-9dfa-b86b61063ae8" path="/var/lib/kubelet/pods/d084ee93-37d1-4c1b-9dfa-b86b61063ae8/volumes"
Oct 31 00:23:10.823040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.823017    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 31 00:23:10.823176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.823121    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" event={"ID":"d46ade2d-3c27-496c-b4c4-d0a80f55f9a0","Type":"ContainerStarted","Data":"596840a00632709bde88533a155c95c339d964ae2667ec331fb7ceb061be9650"}
Oct 31 00:23:10.823176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.823150    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 31 00:23:10.823176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.823167    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" event={"ID":"3c0d9ceb-7c2b-4872-a065-4bfe4b332064","Type":"ContainerStarted","Data":"a6bacee7da38456f3bfa7c9ee672713d8e0300060cf3636d941f16a9dfeddb66"}
Oct 31 00:23:10.969910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.969830    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:10.970382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:10.969906    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.683680    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.684558    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" event={"ID":"578a9d2b-c161-4a91-8f5a-8cf5a891a2bb","Type":"ContainerStarted","Data":"4ce907f53612efcafdf563716ba5a5f3f3aae19a45e2f9931c37d0deb1fd6ab5"}
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.685026    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.685130    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.685183    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.688989    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" event={"ID":"bf1eaf8e-40da-42fe-bb27-6900f2488745","Type":"ContainerStarted","Data":"1c057c016c2ee3459ee62a6164f5ef2b66db55ff1a7063b7f21728a3b72188db"}
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.689381    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.698561    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6wdkk" event={"ID":"b6209fdf-df1c-4a4f-b5ce-55812e9f07a4","Type":"ContainerStarted","Data":"0e5f1b8424f68705d378b8dd1898923e2f658418b3fad580011db11663991074"}
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.698613    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.701287    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.701478    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Oct 31 00:23:11.732932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.722122    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt"
Oct 31 00:23:12.005722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.970606    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:12.005722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:11.970684    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:12.101662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.048541    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"]
Oct 31 00:23:12.101662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.058787    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e480172b-sd5md"]
Oct 31 00:23:12.264739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.264658    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2" path="/var/lib/kubelet/pods/1d729f76-6cc6-4bb4-bb4d-afc3a7e81db2/volumes"
Oct 31 00:23:12.710521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.710069    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Oct 31 00:23:12.710521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.710131    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused"
Oct 31 00:23:12.710521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.710192    5078 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-6wdkk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Oct 31 00:23:12.710521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.710230    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-6wdkk" podUID="b6209fdf-df1c-4a4f-b5ce-55812e9f07a4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Oct 31 00:23:12.970529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.970403    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:12.970529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:12.970505    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:13.217645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.217615    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"]
Oct 31 00:23:13.223567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.223500    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"]
Oct 31 00:23:13.228885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.228864    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"]
Oct 31 00:23:13.247509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.241211    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"]
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.330216    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.330286    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.330314    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.330433    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.330530    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.330572    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.331603    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.331651    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.333329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.331678    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.340102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.340066    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.340210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.340147    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.340210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.340186    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:23:13.376208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.346053    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 00:23:13.712428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.688631    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:13.712428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.688776    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:13.712428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.688870    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:23:13.722120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.722087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" event={"ID":"5a23b543-84bf-4a68-9e80-414b3fdfc4ea","Type":"ContainerStarted","Data":"dd6761d98889d5adef8c8f2ef6911ab65453f3d5de0e4995c93c49889b2d9554"}
Oct 31 00:23:13.723878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.723854    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" event={"ID":"4da9275e-742e-4820-8580-c48195fe6776","Type":"ContainerStarted","Data":"aff4cd8d1a77960882304c93701d82ca99195bc4fbfccaa3a46632473bf1f88c"}
Oct 31 00:23:13.727049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.727023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" event={"ID":"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d","Type":"ContainerStarted","Data":"d90a17c634cdd09ec2f57b1a4cd79e6356e2f8070e2a7a31fb533403579c48f9"}
Oct 31 00:23:13.728654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:13.728627    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" event={"ID":"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7","Type":"ContainerStarted","Data":"30d28d3f69a79851fd610dcbb54c09c0d100d15be1666cd1ec3b50b8a48a349c"}
Oct 31 00:23:14.007056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.006961    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:14.007056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.007017    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:14.043698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.043663    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"]
Oct 31 00:23:14.049396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.049370    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0428ea-sqkdl"]
Oct 31 00:23:14.264674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.264599    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f171633-a74d-4ec9-a7c3-412672ba9f49" path="/var/lib/kubelet/pods/1f171633-a74d-4ec9-a7c3-412672ba9f49/volumes"
Oct 31 00:23:14.732542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.732447    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:14.732741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.732570    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:14.742822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.742779    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" event={"ID":"5a23b543-84bf-4a68-9e80-414b3fdfc4ea","Type":"ContainerStarted","Data":"e3984bf2c70f13c453fed8693fd631f9e17d6ed6f2bf4275e7439ea20ce9466d"}
Oct 31 00:23:14.745922 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.745888    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" event={"ID":"4da9275e-742e-4820-8580-c48195fe6776","Type":"ContainerStarted","Data":"4b3945c907d84a1579b7b3f8fe3e8fb2a8e7914e79f98c3a5e385e9d768cf8f7"}
Oct 31 00:23:14.748882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.748841    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" event={"ID":"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d","Type":"ContainerStarted","Data":"b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"}
Oct 31 00:23:14.750829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.750790    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" event={"ID":"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7","Type":"ContainerStarted","Data":"19cb826b1ec78cdb5567f5c750c2b4a34271a20f8325ad8baab9558b3bbc6714"}
Oct 31 00:23:14.752968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.752944    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn" event={"ID":"6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152","Type":"ContainerStarted","Data":"665a901f787b63297714f41d8bf634c93719adfda6efd8f243e1fd8baf451e50"}
Oct 31 00:23:14.753103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.753088    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:23:14.824175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.824123    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podStartSLOduration=50.824109085 podStartE2EDuration="50.824109085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:14.812784713 +0000 UTC m=+1762.759155444" watchObservedRunningTime="2026-10-31 00:23:14.824109085 +0000 UTC m=+1762.770479855"
Oct 31 00:23:14.868177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.868130    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podStartSLOduration=42.868115905 podStartE2EDuration="42.868115905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:14.854688699 +0000 UTC m=+1762.801059419" watchObservedRunningTime="2026-10-31 00:23:14.868115905 +0000 UTC m=+1762.814486622"
Oct 31 00:23:14.929017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.904744    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podStartSLOduration=50.904724919 podStartE2EDuration="50.904724919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:14.893989669 +0000 UTC m=+1762.840360388" watchObservedRunningTime="2026-10-31 00:23:14.904724919 +0000 UTC m=+1762.851095689"
Oct 31 00:23:14.938052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.937997    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" podStartSLOduration=21.937984136 podStartE2EDuration="21.937984136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:14.931844761 +0000 UTC m=+1762.878215482" watchObservedRunningTime="2026-10-31 00:23:14.937984136 +0000 UTC m=+1762.884354864"
Oct 31 00:23:14.947021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.946997    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:14.948713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.948689    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:23:14.948802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.948732    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:23:14.981513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.979030    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:14.994524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.990537    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:14.994524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.990598    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:14.994524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.992511    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 00:23:14.994524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:14.992566    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 00:23:15.095485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.090704    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:15.095485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.093350    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 00:23:15.095485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.093406    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 00:23:15.502022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.501702    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 00:23:15.502022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.501766    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 00:23:15.502022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.501791    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:15.974491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.969503    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:15.974491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.969566    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:15.990493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.985291    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 00:23:15.990493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.985344    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 00:23:15.990493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:15.985940    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:16.094523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.093261    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 00:23:16.094523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.093320    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 00:23:16.508259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.508226    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:16.864625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.864526    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.864974    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.866550    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:16.866591    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.047109    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.047164    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.047742    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 00:23:17.059354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.047780    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 00:23:17.440242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.146556    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 00:23:17.440242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.146623    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 00:23:17.440242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.434632    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:17.440242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:17.434689    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:20.647666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:20.647620    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:23:21.380713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:20.756822    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 00:23:21.380713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:20.756876    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 00:23:22.236523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:22.236455    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:22.269422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:22.237001    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:22.269422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:22.237530    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:22.269422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:22.258326    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.61:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:22.269422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:22.258447    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.63:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:24.131142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.131107    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.131162    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162671    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162797    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162864    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162901    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162931    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162961    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.162998    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:23:24.236874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.163059    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.353131    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.551542    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.551881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.553409    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.553768    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.576453    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.62:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.576994    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.577042    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.891995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" podUID="fe713694-148e-4e08-bc8a-becc4f002df5" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.893057    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.893134    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.893202    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.893504    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:24.916544    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:25.163534    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.572337    5078 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.572440    5078 patch_prober.go:29] interesting pod/image-registry-6c895f9d65-tkw4d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.572489    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" podUID="2f6a88c2-1453-4033-8027-afc8b69d3ee7" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.572741    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" podUID="c108abe8-dc34-426e-9a4e-becaf4e882b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:26.572779    5078 logging.go:55] [core] [Channel #1558 SubChannel #1564]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.63:50052", ServerName: "10.132.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.63:50052: operation was canceled"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.572792    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:26.583520    5078 logging.go:55] [core] [Channel #1557 SubChannel #1563]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.61:50052", ServerName: "10.132.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.61:50052: operation was canceled"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.583725    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.583779    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.583815    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.583848    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584136    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584448    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584499    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584527    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584576    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584603    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584555    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.613590    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:27.236210    5078 patch_prober.go:29] interesting pod/image-registry-6c895f9d65-tkw4d container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:27.256259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:27.236221    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:27.573832    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:27.894379    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:29.576808    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qnjc2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:29.576891    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.152:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:29.667677    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" podUID="654d9cdc-9ff9-4d6d-96e7-35fe4f90894f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:29.731109    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:29.905667    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:30.120286    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:34.003854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:30.515682    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:43.093643    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:43.343891    5078 patch_prober.go:29] interesting pod/image-registry-6c895f9d65-tkw4d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:43.343952    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" podUID="2f6a88c2-1453-4033-8027-afc8b69d3ee7" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:43.362158    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:43.376291    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": dial tcp 10.132.0.37:8081: connect: connection refused"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.134030    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.276347    5078 trace.go:236] Trace[1888727383]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" (31-Oct-2026 00:23:43.121) (total time: 1154ms):
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: Trace[1888727383]: [1.154540306s] [1.154540306s] END
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.282886    5078 trace.go:236] Trace[1595751458]: "Calculate volume metrics of kube-api-access-9l7j5 for pod opendatahub/ai-gateway-controller-86f7d849b4-vgljn" (31-Oct-2026 00:23:43.079) (total time: 1203ms):
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: Trace[1595751458]: [1.203276145s] [1.203276145s] END
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.435983    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.436036    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.436108    5078 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.436126    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" podUID="c108abe8-dc34-426e-9a4e-becaf4e882b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.575404    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.578091    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.578132    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.578311    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.578333    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.578498    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.583492    5078 patch_prober.go:29] interesting pod/dns-default-shskm container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.583522    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-shskm" podUID="d1384f8a-a2ad-4e55-b6e5-14ab134fa489" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.17:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.639189    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.639224    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.639278    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.639295    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.639327    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.639516    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.163:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:44.639953    5078 logging.go:55] [core] [Channel #1565 SubChannel #1566]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.163:50052", ServerName: "10.132.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.163:50052: i/o timeout"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.643884    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.643915    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.656211    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.656243    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.661961    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.661990    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:44.686887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.667207    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.687527    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.690005    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": dial tcp 10.132.0.37:8081: connect: connection refused"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.697224    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.698224    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.698251    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.721569    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.721601    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.841434    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.841451    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.841523    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.851111    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.852282    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.858570    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:26.584735    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.876432    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.78764691s: [/var/lib/containers/storage/overlay/34ba225eeb5929b82b742fa6610a773baa71504d27a1fe1912befa22309ca733/diff /var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-vqgls_daf52a97-c6ca-4fba-a77b-67841614c162/service-ca-operator/1.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.880446    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c895f9d65-tkw4d" podUID="2f6a88c2-1453-4033-8027-afc8b69d3ee7" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.886715    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.434502665s: [/var/lib/containers/storage/overlay/7537c642c473f9b65b96425084b2d9e0d488b02ba838416a900ec41334bf5f2f/diff /var/log/pods/openshift-ingress_e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq_0ec42ad3-37b6-41b2-930b-5756d51a23b3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.897159    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.441731347s: [/var/lib/containers/storage/overlay/5dcfa660820a53c33fc026b71fcef8fb948ba16d8c8618358a3b5297c38336ec/diff ]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.897592    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.442149165s: [/var/lib/containers/storage/overlay/1f82bae830cabcba9c83cf7f25782740ba18eaa78ca57420b5592bd52dd0c40b/diff /var/log/pods/openshift-monitoring_node-exporter-kpj57_c25f317f-5d84-42c6-b06d-e69dbb04420e/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.938533    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.991988504s: [/var/lib/containers/storage/overlay/f48d866ab361c8acb8794a5d466145f2c16e67f923e88f81e18b2691f207313c/diff ]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.948237    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.166:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.959692    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.504255585s: [/var/lib/containers/storage/overlay/c30220a754584d1dd7a31b031ef40ca0e3b8fad6db7ef577565991444ccf6c06/diff /var/log/pods/openshift-ingress_e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4_0c1d12b5-4ac9-46e9-a368-36f48a923c59/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.960070    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.439141778s: [/var/lib/containers/storage/overlay/a907e177c1524a1d3fb85b5b2013ef4870d0c8cc765dbde245046576aca8987b/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qnjc2_3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8/discovery/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:28.574629    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.973930    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.62:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.975845    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.61:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:44.977628    5078 logging.go:55] [core] [Channel #1569 SubChannel #1599]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.61:50052", ServerName: "10.132.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.61:50052: operation was canceled"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.977686    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.63:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:44.978220    5078 logging.go:55] [core] [Channel #1568 SubChannel #1596]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.62:50052", ServerName: "10.132.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.62:50052: operation was canceled"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:44.979580    5078 logging.go:55] [core] [Channel #1570 SubChannel #1600]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.63:50052", ServerName: "10.132.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.63:50052: operation was canceled"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:44.991610    5078 logging.go:55] [core] [Channel #1567 SubChannel #1591]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.166:50052", ServerName: "10.132.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:44914->10.132.0.166:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.993951    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.047417679s: [/var/lib/containers/storage/overlay/75d4aba94d540f4464bf0a35d697e7a382f10cf52cf941732da95aff582c8338/diff /var/log/pods/openshift-ingress_e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc_75bc669f-f382-41b1-8a91-b79e0b976575/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.997867    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:44.998306    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c2w2g" podUID="fe713694-148e-4e08-bc8a-becc4f002df5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.55:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:44.998558    5078 logging.go:55] [core] [Channel #1571 SubChannel #1608]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.55:50052", ServerName: "10.132.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.55:50052: operation was canceled"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.000002    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qnjc2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.000076    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.152:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.003710    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.544087863s: [/var/lib/containers/storage/overlay/876622d64dc6ccbfefe34a3bd4780ca7e584861c1c1c8f17c8954d15eed9ca81/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824_314a02c0-fdfd-4053-a213-5b8691190a5a/maas-api/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.004068    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.005948    5078 logging.go:55] [core] [Channel #1572 SubChannel #1612]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.160:50052: operation was canceled"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.014811    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.240994602s: [/var/lib/containers/storage/overlay/91ca6b278e0d098f9fdb6cc0fab978faed6601b8b7e4e51cf7412865093225ee/diff /var/log/pods/openshift-ingress_e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl_9d43026d-495b-4d5c-b30c-4ee2f68bb8f4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.084962    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.151726823s: [/var/lib/containers/storage/overlay/a9970f7e33e07c4b595f05ebf18a1629ba90f22522cfa3e9f74becb418394f5f/diff /var/log/pods/openshift-ingress_e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd_3ad9fb99-b725-4f80-b139-ceaba025da2c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.111223    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.658994502s: [/var/lib/containers/storage/overlay/20392e5574321cb8c03d8658daf961a924993c72d060ac1a39432d0d5193b231/diff /var/log/pods/openshift-ingress_e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q_aa8de2cb-4f63-4998-93a9-aff00d53183e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.128952    5078 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 15.592879153s: [/var/lib/containers/storage/overlay/e2c5fb76496b1435e3b4e9f99d8f4663ba74e39cc517db9f0a4e51cd7b8bcc5d/diff /var/log/pods/e2e-models-e2e-worker-w5-7b2341_simulator-w5-7b2341-kserve-c87fb5b59-svr9n_0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f/main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.611210    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.611599    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.611648    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.611754    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.66:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.624209    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.626922    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.628984    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.643143    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.643214    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.643494    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.163:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.645314    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.645368    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.648198    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.648252    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.650880    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.650918    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.653680    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.653718    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.653746    5078 logging.go:55] [core] [Channel #1577 SubChannel #1578]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.164:50052", ServerName: "10.132.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:36796->10.132.0.164:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.662612    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.662680    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.663657    5078 logging.go:55] [core] [Channel #1575 SubChannel #1576]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.66:50052", ServerName: "10.132.0.66:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:45968->10.132.0.66:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.664691    5078 logging.go:55] [core] [Channel #1573 SubChannel #1574]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51170->10.132.0.64:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.667873    5078 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.667909    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b" podUID="922f74f1-f495-4764-b2c9-1f40b2bcf849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.677560    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.677599    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.677827    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.65:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.677985    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.678010    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.678401    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.678435    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.682106    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.682189    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.684057    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.684092    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.691706    5078 logging.go:55] [core] [Channel #1582 SubChannel #1583]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.163:50052", ServerName: "10.132.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:49060->10.132.0.163:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.697385    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.697417    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.704432    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.705674    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.705703    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.705922    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.705968    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.709660    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.709692    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.718639    5078 logging.go:55] [core] [Channel #1585 SubChannel #1586]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.65:50052", ServerName: "10.132.0.65:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52082->10.132.0.65:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:23:45.724725    5078 logging.go:55] [core] [Channel #1588 SubChannel #1589]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.67:50052", ServerName: "10.132.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:42644->10.132.0.67:50052: use of closed network connection"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.745557    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.844017    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.844075    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.844169    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.851376    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.851413    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.856936    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.856985    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.862353    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.862392    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.864892    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.864927    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.898648    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.898677    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.899588    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.899616    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.910496    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.910534    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.949490    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.949537    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.951128    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.951163    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.962037    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.962065    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.971217    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.971266    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.971416    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.971527    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.981107    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:45.981135    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.009579    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.009614    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.017584    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 00:23:46.098932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.094276    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.343951    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.343995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.664201    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.664253    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.684502    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.684545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.704568    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.704607    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.705563    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.705590    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.786548    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.941697    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:46.941755    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.050594    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.050658    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.320577    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.320639    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:47.554392    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.925s"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.554521    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.554575    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.676978    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.678083    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.678169    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.678244    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.678294    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.679590    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680643    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680693    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680723    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe0ee14-mftlj"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680757    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680777    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680805    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229a4a6-5snnb"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680847    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680892    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680913    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680932    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc93e3e-grmsx"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680948    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.680985    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681004    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c82dab9-hvhx5"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681040    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681054    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681070    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681108    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86f7d849b4-vgljn"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681149    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681166    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40faa617-rq8d2"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681205    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681237    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681254    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218254ba-gdbx8"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681270    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681304    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681320    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0a1b16-8jt8c"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681354    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681370    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681386    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681409    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681447    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681485    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681506    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f7613-dhn5n"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681522    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681538    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681567    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681580    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e430b93-jw6mr"]
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681616    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681672    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.681810    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.682300    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.686957    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687003    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687025    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687061    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687101    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6wdkk"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687137    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687167    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687181    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687219    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687241    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.687284    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.697593    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.709849    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.709912    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.745613    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.745657    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.943203    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.943249    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.965587    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.965640    5078 generic.go:356] "Generic (PLEG): container finished" podID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerID="6a8f117cb1730c3a9d8dd12471c1758fe1e91086c7d36273454a6d9e1a23fd4b" exitCode=137
Oct 31 00:23:48.022609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:47.965710    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74cc75bf99-fkb87" event={"ID":"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1","Type":"ContainerDied","Data":"6a8f117cb1730c3a9d8dd12471c1758fe1e91086c7d36273454a6d9e1a23fd4b"}
Oct 31 00:23:49.557218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:48.159619    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:49.557218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:48.159720    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:49.557218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:48.709677    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15489863-c603-4bd2-8d51-7e90cdf15244" path="/var/lib/kubelet/pods/15489863-c603-4bd2-8d51-7e90cdf15244/volumes"
Oct 31 00:23:49.557218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:49.065115    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546fbce6-4e3d-4829-bc58-cb72086abd10" path="/var/lib/kubelet/pods/546fbce6-4e3d-4829-bc58-cb72086abd10/volumes"
Oct 31 00:23:49.557218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:49.429374    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Oct 31 00:23:49.557218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:49.548412    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" podUID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": dial tcp 10.132.0.37:8081: connect: connection refused"
Oct 31 00:23:49.719140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:49.594295    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8640a5d5-91f3-40c4-b062-98dfedb6f1e6" path="/var/lib/kubelet/pods/8640a5d5-91f3-40c4-b062-98dfedb6f1e6/volumes"
Oct 31 00:23:50.039329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.022795    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4d04b6-98ea-41cd-8ecd-77ca78966a01" path="/var/lib/kubelet/pods/8a4d04b6-98ea-41cd-8ecd-77ca78966a01/volumes"
Oct 31 00:23:50.192157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.192073    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ef0771-1298-435d-82f8-7716d0feb818" path="/var/lib/kubelet/pods/91ef0771-1298-435d-82f8-7716d0feb818/volumes"
Oct 31 00:23:50.359039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.359006    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f9f64f-e770-4c36-aad8-6975db123241" path="/var/lib/kubelet/pods/b2f9f64f-e770-4c36-aad8-6975db123241/volumes"
Oct 31 00:23:50.361588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.361561    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:50.361711 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.361620    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:50.448421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.448360    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba29423d-2585-4527-bd0f-ceee7ec96593" path="/var/lib/kubelet/pods/ba29423d-2585-4527-bd0f-ceee7ec96593/volumes"
Oct 31 00:23:50.729062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.728977    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e9934d-8faf-4370-9c19-01805736a20d" path="/var/lib/kubelet/pods/c6e9934d-8faf-4370-9c19-01805736a20d/volumes"
Oct 31 00:23:50.786607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.786566    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:50.786780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.786626    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:50.853361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.853332    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f640ab64-980a-490e-a79b-43bcee5fc7ca" path="/var/lib/kubelet/pods/f640ab64-980a-490e-a79b-43bcee5fc7ca/volumes"
Oct 31 00:23:50.949578 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:50.949546    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.763s"
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.949665    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.949725    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.979246    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"]
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.979286    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9285ec5-4xqtc"]
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.979310    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"]
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:50.979325    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab88dc0-nww8b"]
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.012180    5078 generic.go:356] "Generic (PLEG): container finished" podID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerID="e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029" exitCode=0
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.012249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" event={"ID":"7e84018f-f7a2-4588-bd5b-b67814c509bd","Type":"ContainerDied","Data":"e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"}
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.012932    5078 scope.go:122] "RemoveContainer" containerID="e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.014780    5078 generic.go:356] "Generic (PLEG): container finished" podID="e9bb7055-cf51-4182-a092-2364ed3d5444" containerID="dbd38a7138471dca14bc467e687c3f54e74c3572af13eaf29768ff1418d1eb12" exitCode=1
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.014849    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" event={"ID":"e9bb7055-cf51-4182-a092-2364ed3d5444","Type":"ContainerDied","Data":"dbd38a7138471dca14bc467e687c3f54e74c3572af13eaf29768ff1418d1eb12"}
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.016706    5078 generic.go:356] "Generic (PLEG): container finished" podID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerID="27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e" exitCode=1
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.016775    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" event={"ID":"68cb91e9-c6e4-460f-9318-49d79080dedb","Type":"ContainerDied","Data":"27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"}
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.017248    5078 scope.go:122] "RemoveContainer" containerID="27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.041244    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"]
Oct 31 00:23:51.060138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.050823    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befaf1e5-q4hm7"]
Oct 31 00:23:51.133828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.063517    5078 scope.go:122] "RemoveContainer" containerID="dbd38a7138471dca14bc467e687c3f54e74c3572af13eaf29768ff1418d1eb12"
Oct 31 00:23:51.234298 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:23:51.234263    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:23:51.641171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:51.640415    5078 trace.go:236] Trace[1038748856]: "Calculate volume metrics of manifests for pod opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" (31-Oct-2026 00:23:43.463) (total time: 8177ms):
Oct 31 00:23:51.641171 ip-10-0-143-16 kubenswrapper[5078]: Trace[1038748856]: [8.177321806s] [8.177321806s] END
Oct 31 00:23:52.089445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:52.086793    5078 trace.go:236] Trace[1850152734]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-tm95b" (31-Oct-2026 00:23:44.703) (total time: 7383ms):
Oct 31 00:23:52.089445 ip-10-0-143-16 kubenswrapper[5078]: Trace[1850152734]: [7.383691807s] [7.383691807s] END
Oct 31 00:23:52.286338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:52.286250    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f26466-4fc1-4b05-bda6-5b0cc29d4eb6" path="/var/lib/kubelet/pods/19f26466-4fc1-4b05-bda6-5b0cc29d4eb6/volumes"
Oct 31 00:23:52.816840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:52.816790    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497275d4-4d81-4d69-8c6b-b3daec08ffc0" path="/var/lib/kubelet/pods/497275d4-4d81-4d69-8c6b-b3daec08ffc0/volumes"
Oct 31 00:23:52.979531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:52.979490    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5722e205-4aab-4446-b1ea-a244b5f3da75" path="/var/lib/kubelet/pods/5722e205-4aab-4446-b1ea-a244b5f3da75/volumes"
Oct 31 00:23:53.032823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.032797    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:23:53.033028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.032806    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:23:53.045367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.045332    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v" event={"ID":"e9bb7055-cf51-4182-a092-2364ed3d5444","Type":"ContainerStarted","Data":"3b47d316ea9aa099f8d4812f65861802a9b55304bdc3fbb0ffc7dd74fc9e94f1"}
Oct 31 00:23:53.045789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.045766    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.047536    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.047620    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74cc75bf99-fkb87" event={"ID":"d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1","Type":"ContainerStarted","Data":"5e7e36480d55adb36d2d57f2e92da53e94e994fa55d77d54f8bb8a8691bb3255"}
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.047775    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.047987    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused" start-of-body=
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.048030    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="Get \"http://10.132.0.20:1936/healthz/ready\": dial tcp 10.132.0.20:1936: connect: connection refused"
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.049976    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" event={"ID":"7e84018f-f7a2-4588-bd5b-b67814c509bd","Type":"ContainerStarted","Data":"38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634"}
Oct 31 00:23:53.078364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.050180    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:23:53.259386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.259358    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 31 00:23:53.888805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.888774    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:23:53.965587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:53.888774    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:23:54.396000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:54.395963    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 31 00:23:54.396000 ip-10-0-143-16 kubenswrapper[5078]: [+]has-synced ok
Oct 31 00:23:54.396000 ip-10-0-143-16 kubenswrapper[5078]: [+]process-running ok
Oct 31 00:23:54.396000 ip-10-0-143-16 kubenswrapper[5078]: healthz check failed
Oct 31 00:23:54.396771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:54.396027    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 31 00:23:54.652563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:54.650631    5078 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-68c79 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:54.652563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:54.650696    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68c79" podUID="6b213ff9-3e00-426a-901f-e5ca105d37e8" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:54.840607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:54.840563    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:54.840819 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:54.840616    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:23:55.124785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:55.124715    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"]
Oct 31 00:23:55.152496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:55.152450    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d8259-8ptng"]
Oct 31 00:23:55.259851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:55.259824    5078 patch_prober.go:29] interesting pod/router-default-74cc75bf99-fkb87 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 31 00:23:55.259851 ip-10-0-143-16 kubenswrapper[5078]: [+]has-synced ok
Oct 31 00:23:55.259851 ip-10-0-143-16 kubenswrapper[5078]: [+]process-running ok
Oct 31 00:23:55.259851 ip-10-0-143-16 kubenswrapper[5078]: healthz check failed
Oct 31 00:23:55.260086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:55.259869    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-74cc75bf99-fkb87" podUID="d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 31 00:23:55.959957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:55.959930    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:23:56.271064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:56.224327    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f711ec-24e6-4e9d-9d42-abb2d9b65f92" path="/var/lib/kubelet/pods/61f711ec-24e6-4e9d-9d42-abb2d9b65f92/volumes"
Oct 31 00:23:56.271064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:56.260439    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 31 00:23:56.271064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:56.261912    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74cc75bf99-fkb87"
Oct 31 00:23:57.099280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:57.099238    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" event={"ID":"68cb91e9-c6e4-460f-9318-49d79080dedb","Type":"ContainerStarted","Data":"c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a"}
Oct 31 00:23:57.099764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:57.099568    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:23:59.730603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:59.730563    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:23:59.734231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:23:59.730622    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:03.633019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:03.632985    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"]
Oct 31 00:24:03.633584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:03.633225    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" containerID="cri-o://d8ac09d1e03aa3d9c57a2c8d467120b068d0b5e3e223f94521eebe400c67967e" gracePeriod=30
Oct 31 00:24:03.712764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:03.638534    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"]
Oct 31 00:24:03.712764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:03.638702    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" containerID="cri-o://715907364b15dc74f4b24741a16ee080ca2da1326f021eccf9dee409973fd1e0" gracePeriod=30
Oct 31 00:24:04.041278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:04.029177    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"]
Oct 31 00:24:04.041278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:04.034379    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfeb759-dd88b"]
Oct 31 00:24:04.263970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:04.263933    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713171b0-70e3-4006-9ab6-86e6d5c72505" path="/var/lib/kubelet/pods/713171b0-70e3-4006-9ab6-86e6d5c72505/volumes"
Oct 31 00:24:04.326893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:04.326822    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:24:06.397850 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:06.397807    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:24:07.167565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:07.167516    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 00:24:07.167752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:07.167589    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:24:08.111187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:08.111150    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:24:10.112783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:09.840603    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:10.112783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:09.840657    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:10.247148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:10.247114    5078 generic.go:356] "Generic (PLEG): container finished" podID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerID="715907364b15dc74f4b24741a16ee080ca2da1326f021eccf9dee409973fd1e0" exitCode=0
Oct 31 00:24:10.247338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:10.247187    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" event={"ID":"0ec42ad3-37b6-41b2-930b-5756d51a23b3","Type":"ContainerDied","Data":"715907364b15dc74f4b24741a16ee080ca2da1326f021eccf9dee409973fd1e0"}
Oct 31 00:24:10.311329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:10.248900    5078 generic.go:356] "Generic (PLEG): container finished" podID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerID="d8ac09d1e03aa3d9c57a2c8d467120b068d0b5e3e223f94521eebe400c67967e" exitCode=0
Oct 31 00:24:10.311329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:10.248970    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" event={"ID":"9cedc8c4-4898-447c-8f36-75c1ab327b7a","Type":"ContainerDied","Data":"d8ac09d1e03aa3d9c57a2c8d467120b068d0b5e3e223f94521eebe400c67967e"}
Oct 31 00:24:12.434062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:12.434011    5078 scope.go:122] "RemoveContainer" containerID="7997537535a8ad28ec5143a91c1eb9c22bfac5c93dda297b6255e833b737132f"
Oct 31 00:24:13.589011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:13.588834    5078 scope.go:122] "RemoveContainer" containerID="374131d55b8cb84713c86de99521fd24a4833fa16ab1a514d8419fde6918e103"
Oct 31 00:24:13.691615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:13.691581    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 00:24:13.691787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:13.691658    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 00:24:14.069572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:14.069537    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mpl9v"
Oct 31 00:24:14.232063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:14.232039    5078 scope.go:122] "RemoveContainer" containerID="a06149d77879cae310393704e73664d155ed8c36322387ba5f6c525a55cebbd5"
Oct 31 00:24:14.731635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:14.731585    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:14.732141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:14.731649    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:15.256942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:15.256728    5078 scope.go:122] "RemoveContainer" containerID="ed36dbb828c436d91311aac26b78bfe9f0d29c3af752b57aa64603622c20d8ad"
Oct 31 00:24:16.602786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:16.602699    5078 scope.go:122] "RemoveContainer" containerID="b57a345318ef1a992828c427448522bf21a9f3cd277f300068019935f45cfb1a"
Oct 31 00:24:17.037915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:17.037883    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"]
Oct 31 00:24:17.043360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:17.043336    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e29a7d9-7r7n5"]
Oct 31 00:24:17.327686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:17.327425    5078 scope.go:122] "RemoveContainer" containerID="d6f22c8947f4ba632b4c905562014ffc057985cf4c0554e3b8ce78979819ee78"
Oct 31 00:24:18.200985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:18.200944    5078 scope.go:122] "RemoveContainer" containerID="2b1c8341be38bd226248a87a71449b68631fb0f10eb0a625b3e7d1f7121ac158"
Oct 31 00:24:18.237800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:18.237770    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:18.238602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:18.238569    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:18.585979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:18.585939    5078 scope.go:122] "RemoveContainer" containerID="c36ad0f04cdebee4c5d7c678bc33aa9d1753cf6a72a050271bcd6be3c11740af"
Oct 31 00:24:18.917152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:18.917068    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7" path="/var/lib/kubelet/pods/68ed62e6-c3fb-44b7-b0ad-2733b5a5c4d7/volumes"
Oct 31 00:24:19.244817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.244792    5078 scope.go:122] "RemoveContainer" containerID="a2373afc0eedb17a210ae8f2da2df629150dc2a353f14856f4e41641774cd329"
Oct 31 00:24:19.342972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.342952    5078 scope.go:122] "RemoveContainer" containerID="6d6b23c55690a60b85350b6c80f98d020a1f96cb17625c8d3abcaf3a6ad7abeb"
Oct 31 00:24:19.718662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.568856    5078 scope.go:122] "RemoveContainer" containerID="11d7f77fce6632eeb30dd9cd9b2ff6a5ac3c873d7382a0045e223c18a626f0cf"
Oct 31 00:24:19.813700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.813666    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"]
Oct 31 00:24:19.815038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.815013    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c7e5fd92-5df1-44cc-a210-1e1534d35281" containerName="revoke-keys"
Oct 31 00:24:19.815038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.815037    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e5fd92-5df1-44cc-a210-1e1534d35281" containerName="revoke-keys"
Oct 31 00:24:19.815425 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:19.815403    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e5fd92-5df1-44cc-a210-1e1534d35281" containerName="revoke-keys"
Oct 31 00:24:22.238638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:22.166408    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 00:24:22.238638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:22.166478    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 00:24:22.585538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:22.135270    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:22.920880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:22.920837    5078 patch_prober.go:29] interesting pod/jobset-controller-manager-657fcc8b8f-z9psz container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:23.165775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:22.920981    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-657fcc8b8f-z9psz" podUID="3c0d9ceb-7c2b-4872-a065-4bfe4b332064" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:23.413518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.390230    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:23.413518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.390365    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:23.413518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.390634    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kd9\" (UniqueName: \"kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.480117    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.480117    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.495655    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.495703    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.495743    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kd9\" (UniqueName: \"kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.723048    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:24.046761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:23.972335    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ljnt9" podUID="bf1eaf8e-40da-42fe-bb27-6900f2488745" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:24.347246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.240655    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"]
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.393525    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.600521    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.609226    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-719e12-serving-cert\""
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.732931    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.732978    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jl6x\" (UniqueName: \"kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.733190    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.733245    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.835041    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.835100    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jl6x\" (UniqueName: \"kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.835367    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.835499    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:24.835522    5078 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-serving-cert: secret "maas-api-e2e-api-a-719e12-serving-cert" not found
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:24.835588    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls podName:b5c1b29b-5502-46d7-bade-e05d3be82fe0 nodeName:}" failed. No retries permitted until 2026-10-31 00:24:25.335571136 +0000 UTC m=+1833.281941866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls") pod "maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0") : secret "maas-api-e2e-api-a-719e12-serving-cert" not found
Oct 31 00:24:25.155887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:24.835765    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.832796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:25.356222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:25.832796 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:25.356578    5078 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-serving-cert: secret "maas-api-e2e-api-a-719e12-serving-cert" not found
Oct 31 00:24:25.832796 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:25.356641    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls podName:b5c1b29b-5502-46d7-bade-e05d3be82fe0 nodeName:}" failed. No retries permitted until 2026-10-31 00:24:26.356624243 +0000 UTC m=+1834.302994953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls") pod "maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0") : secret "maas-api-e2e-api-a-719e12-serving-cert" not found
Oct 31 00:24:25.832796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:25.735866    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"]
Oct 31 00:24:25.832796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:25.746874    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"]
Oct 31 00:24:26.479911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.479607    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:26.479911 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:26.479821    5078 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-serving-cert: secret "maas-api-e2e-api-a-719e12-serving-cert" not found
Oct 31 00:24:26.499810 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:26.480711    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls podName:b5c1b29b-5502-46d7-bade-e05d3be82fe0 nodeName:}" failed. No retries permitted until 2026-10-31 00:24:28.479876167 +0000 UTC m=+1836.426246876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls") pod "maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0") : secret "maas-api-e2e-api-a-719e12-serving-cert" not found
Oct 31 00:24:26.880025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.716507    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq" event={"ID":"0ec42ad3-37b6-41b2-930b-5756d51a23b3","Type":"ContainerDied","Data":"01f410f494180c9195d240e3d4da65d932515e7d553117ebff1b455bb0fcaefb"}
Oct 31 00:24:26.880025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.716570    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f410f494180c9195d240e3d4da65d932515e7d553117ebff1b455bb0fcaefb"
Oct 31 00:24:26.880025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.717880    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg" event={"ID":"9cedc8c4-4898-447c-8f36-75c1ab327b7a","Type":"ContainerDied","Data":"0c7aed3a2968ef58bc782382322c9659202d750ee3e15c82889635d9d0c537ea"}
Oct 31 00:24:26.880025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.717915    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7aed3a2968ef58bc782382322c9659202d750ee3e15c82889635d9d0c537ea"
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.952333    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.989859    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.989919    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990109    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990201    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990241    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990341    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990366    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990599    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.051025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:26.990727    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert\") pod \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\" (UID: \"0ec42ad3-37b6-41b2-930b-5756d51a23b3\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.100708    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.105878    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.105951    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.116299    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.116701    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.117098    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kd9\" (UniqueName: \"kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9\") pod \"maas-api-revoke-sub-f1ede3ba-ps7nr\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.119338    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.119365    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.119381    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.132156    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jl6x\" (UniqueName: \"kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.168256    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.169670    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.224115    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.224146    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.310020    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.335780    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.335966    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336039    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336213    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336279    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336361    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336519    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.336895    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9\") pod \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\" (UID: \"9cedc8c4-4898-447c-8f36-75c1ab327b7a\") "
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.356697    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.360268    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.361974    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.363771    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "kube-api-access-qjgn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.363837    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.363923    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.364127    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy" pod "0ec42ad3-37b6-41b2-930b-5756d51a23b3" (UID: "0ec42ad3-37b6-41b2-930b-5756d51a23b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.368698    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.369158    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.380668    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "kube-api-access-s7wj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.414522    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.414612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.419333    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.438595    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"]
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.438697    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy" pod "9cedc8c4-4898-447c-8f36-75c1ab327b7a" (UID: "9cedc8c4-4898-447c-8f36-75c1ab327b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.439669    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.439685    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.439719    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.439724    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.439928    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" containerName="istio-proxy"
Oct 31 00:24:27.447232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.439947    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" containerName="istio-proxy"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456698    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wj9\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-kube-api-access-s7wj9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456726    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456742    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456755    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456769    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456782    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgn9\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-kube-api-access-qjgn9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456795    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456809    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456823    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456836    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456850    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cedc8c4-4898-447c-8f36-75c1ab327b7a-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456863    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cedc8c4-4898-447c-8f36-75c1ab327b7a-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.456877    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec42ad3-37b6-41b2-930b-5756d51a23b3-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.741866    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.745190    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.748514    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"]
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.748640    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.757331    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-34b786fc-serving-cert\""
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.771131    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.771310    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrfq\" (UniqueName: \"kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.771556    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.771612    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.873352    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.873416    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.873594    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.873704    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrfq\" (UniqueName: \"kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.874839    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.923364    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"]
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.928245    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-zvdlq"]
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.941347    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"]
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.947527    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-w99zg"]
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.964806    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.964869    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.234588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:27.967369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrfq\" (UniqueName: \"kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq\") pod \"maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.756962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:28.279944    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec42ad3-37b6-41b2-930b-5756d51a23b3" path="/var/lib/kubelet/pods/0ec42ad3-37b6-41b2-930b-5756d51a23b3/volumes"
Oct 31 00:24:28.756962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:28.280582    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cedc8c4-4898-447c-8f36-75c1ab327b7a" path="/var/lib/kubelet/pods/9cedc8c4-4898-447c-8f36-75c1ab327b7a/volumes"
Oct 31 00:24:28.756962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:28.373602    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:28.756962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:28.486271    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:28.756962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:28.505131    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"maas-api-e2e-api-a-719e12-75bd5c569b-67g2j\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:28.756962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:28.582968    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:29.762486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:29.754006    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:29.762486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:29.754079    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:31.458674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.435961    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"]
Oct 31 00:24:31.622541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.622503    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"]
Oct 31 00:24:31.622922 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.622903    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:31.653843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.640749    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-945abd-dockercfg-f89qt\""
Oct 31 00:24:31.662190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.661651    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-945abd\""
Oct 31 00:24:31.669124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.665181    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:31.669124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.665448    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmm48\" (UniqueName: \"kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48\") pod \"payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:31.781218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.781106    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmm48\" (UniqueName: \"kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48\") pod \"payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:31.781979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.781947    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:31.803008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.802979    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"]
Oct 31 00:24:31.968494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:31.963310    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:32.177169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.165672    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"]
Oct 31 00:24:32.177169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.165813    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.296821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.296787    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.297155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.297132    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttbh\" (UniqueName: \"kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh\") pod \"payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.402488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.369597    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmm48\" (UniqueName: \"kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48\") pod \"payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:32.421512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.411227    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.421832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.421802    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttbh\" (UniqueName: \"kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh\") pod \"payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.446545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.430714    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.740710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.728834    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttbh\" (UniqueName: \"kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh\") pod \"payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:32.811008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.785154    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:32.850666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.813536    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:32.850666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.815213    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:24:32.886689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.871303    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:24:32.886689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:32.873999    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:33.004125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.004014    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"]
Oct 31 00:24:33.178578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.168898    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:33.256618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.255901    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:33.265164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.262834    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-719e12\""
Oct 31 00:24:33.265433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.265396    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-719e12-dockercfg-s7nkh\""
Oct 31 00:24:33.277061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.276568    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:33.277061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.276622    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:33.366523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.365116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:33.366523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.365165    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dknxx\" (UniqueName: \"kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx\") pod \"payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:33.408415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.408371    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"]
Oct 31 00:24:33.621740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.488369    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:33.621740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.488413    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dknxx\" (UniqueName: \"kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx\") pod \"payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:33.621740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.500539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.843174    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.847541    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.847932    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.855545    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.855749    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.855929    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.856074    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:33.944559    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:34.298904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:34.058634    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:24:34.360827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:34.358135    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:24:34.618543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:34.615826    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.019275    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.096729    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.204945    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.262935    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.274952    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.280304    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.330661    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.336915    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.381038    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dknxx\" (UniqueName: \"kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx\") pod \"payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:35.394269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.383674    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.655895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.595285    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:35.845590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.771942    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:35.967614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:35.860189    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.220685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.176209    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:24:36.220685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.176305    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:24:36.220685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.177501    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.316151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.290772    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.316151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.299028    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.316151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.299164    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:24:36.346883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.346774    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.358574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.346993    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.551534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.541984    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.551534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.542621    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 31 00:24:36.551534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.542780    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:24:36.551534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.543191    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Oct 31 00:24:36.590165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.589651    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.799067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.798964    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:36.801927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.801877    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:24:36.870256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.860732    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.897514    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:36.911170    5078 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.055446    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv" podUID="c108abe8-dc34-426e-9a4e-becaf4e882b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:24:37.023519    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563cf96_6d7c_4dc9_8911_710b77014179.slice/crio-95b35762141381fb996293d94eb676ef9d28b36d3847480941aa4e4c32ac1859 WatchSource:0}: Error finding container 95b35762141381fb996293d94eb676ef9d28b36d3847480941aa4e4c32ac1859: Status 404 returned error can't find the container with id 95b35762141381fb996293d94eb676ef9d28b36d3847480941aa4e4c32ac1859
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:24:37.090815    5078 logging.go:55] [core] [Channel #1811 SubChannel #1812]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.67:50052", ServerName: "10.132.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:52950->10.132.0.67:50052: use of closed network connection"
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.099566    5078 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zzxg7 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.16:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:37.109783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.099623    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zzxg7" podUID="578a9d2b-c161-4a91-8f5a-8cf5a891a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:37.394172    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.036s"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523542    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523586    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" event={"ID":"b5c1b29b-5502-46d7-bade-e05d3be82fe0","Type":"ContainerStarted","Data":"16e7aeb860f1f7934293e57fc68938f7da26f96d9f107c3d8d6212c4d1addf59"}
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523635    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"]
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523667    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr" event={"ID":"1563cf96-6d7c-4dc9-8911-710b77014179","Type":"ContainerStarted","Data":"95b35762141381fb996293d94eb676ef9d28b36d3847480941aa4e4c32ac1859"}
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523691    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" event={"ID":"0ac910e3-9861-4a2a-b293-e7ceb5df4e35","Type":"ContainerStarted","Data":"20ef0c350f5e8d84686fb7370d751a96d21098b083a4d3fa8f106d7bda352a71"}
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523718    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"]
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523877    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523911    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523943    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.523976    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524009    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524027    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"]
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524069    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524102    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524137    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524165    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524194    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524222    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524246    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524271    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524295    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524323    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524350    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524375    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524401    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:24:37.539628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.524427    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:24:37.744733 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:24:37.744669    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b6ab70_6e9a_49af_ac30_8aebecc4b64d.slice/crio-4331a750e8a5b4ae2effb84d8be46ccf84976355790f67a131712cd7cbe26540 WatchSource:0}: Error finding container 4331a750e8a5b4ae2effb84d8be46ccf84976355790f67a131712cd7cbe26540: Status 404 returned error can't find the container with id 4331a750e8a5b4ae2effb84d8be46ccf84976355790f67a131712cd7cbe26540
Oct 31 00:24:37.941504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:37.939374    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"]
Oct 31 00:24:38.240283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:38.240257    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"]
Oct 31 00:24:38.395771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:38.264506    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"]
Oct 31 00:24:38.534979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:38.530605    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp" event={"ID":"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6","Type":"ContainerStarted","Data":"d235728b3fae84b330898aed00533874966ceba7aea5c4372ead4ed11c6180a4"}
Oct 31 00:24:38.534979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:38.532167    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn" event={"ID":"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab","Type":"ContainerStarted","Data":"d09fc0bb3e052a2e8c9995dae31ccb6dbac9a364e18eaf8a2dbd2757380087b7"}
Oct 31 00:24:38.534979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:38.533424    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr" event={"ID":"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d","Type":"ContainerStarted","Data":"4331a750e8a5b4ae2effb84d8be46ccf84976355790f67a131712cd7cbe26540"}
Oct 31 00:24:40.337794 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:24:40.337756    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:24:41.228663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:41.228576    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:41.228663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:41.228638    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:42.847958 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:42.847922    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr" event={"ID":"1563cf96-6d7c-4dc9-8911-710b77014179","Type":"ContainerStarted","Data":"9c4985213ba78c3c7d29f81c56ef3c77326dae5b7c8ee00602e161481abbb144"}
Oct 31 00:24:42.850555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:42.850522    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn" event={"ID":"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab","Type":"ContainerStarted","Data":"7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa"}
Oct 31 00:24:42.976105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:42.853347    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr" event={"ID":"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d","Type":"ContainerStarted","Data":"4d3324909ec1f4b91c968d4cb4733abf056a8c5604279a6401905818852f38db"}
Oct 31 00:24:42.976105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:42.879445    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp" event={"ID":"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6","Type":"ContainerStarted","Data":"45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a"}
Oct 31 00:24:42.976105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:42.899504    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr" podStartSLOduration=23.89947975 podStartE2EDuration="23.89947975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:42.873885673 +0000 UTC m=+1850.820256391" watchObservedRunningTime="2026-10-31 00:24:42.89947975 +0000 UTC m=+1850.845850463"
Oct 31 00:24:43.972318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:43.972277    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:43.997308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:43.997260    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn" podStartSLOduration=12.997244974000001 podStartE2EDuration="12.997244974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:43.992745286 +0000 UTC m=+1851.939116018" watchObservedRunningTime="2026-10-31 00:24:43.997244974 +0000 UTC m=+1851.943615691"
Oct 31 00:24:44.018245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:44.018199    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp" podStartSLOduration=13.018188718 podStartE2EDuration="13.018188718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:44.013074604 +0000 UTC m=+1851.959445341" watchObservedRunningTime="2026-10-31 00:24:44.018188718 +0000 UTC m=+1851.964559436"
Oct 31 00:24:44.036546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:44.036455    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr" podStartSLOduration=13.036442115 podStartE2EDuration="13.036442115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:44.035555494 +0000 UTC m=+1851.981926213" watchObservedRunningTime="2026-10-31 00:24:44.036442115 +0000 UTC m=+1851.982812831"
Oct 31 00:24:45.108611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:45.108573    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:48.447109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:48.442530    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr" event={"ID":"1563cf96-6d7c-4dc9-8911-710b77014179","Type":"ContainerDied","Data":"9c4985213ba78c3c7d29f81c56ef3c77326dae5b7c8ee00602e161481abbb144"}
Oct 31 00:24:48.450636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:48.450552    5078 generic.go:356] "Generic (PLEG): container finished" podID="1563cf96-6d7c-4dc9-8911-710b77014179" containerID="9c4985213ba78c3c7d29f81c56ef3c77326dae5b7c8ee00602e161481abbb144" exitCode=0
Oct 31 00:24:48.499742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:48.499702    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:48.617741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:48.499817    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:48.977666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:48.977611    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:49.031234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:48.977743    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.502834    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-67244" podUID="dc6821bd-245e-4dd1-a279-d34358f5d3a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.502845    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.503421    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.503496    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.503730    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.504661    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.504908    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.505144    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk" podUID="ee668f0d-fda1-4731-83cf-954fa3780710" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.505531    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.506584    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.506665    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv" podUID="3307b72b-996a-4b66-823c-e4347c25e18f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.506930    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.507509    5078 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-94hrt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.507548    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-94hrt" podUID="d46ade2d-3c27-496c-b4c4-d0a80f55f9a0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.507556    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.507562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.508939    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.511288    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.563279    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"]
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.563332    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"]
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.563358    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"]
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.563382    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"]
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.563564    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy" containerID="cri-o://e7a4ea26dd6c40c4199c793c182c33b26766bbf6e67ee9f06e082a3d7d7db1be" gracePeriod=30
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.563899    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy" containerID="cri-o://177dc1749989201340a07dbf9f171c0ba08ec26081e87ece26acc3a00831710b" gracePeriod=30
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.564052    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" containerID="cri-o://00a6f8ab94bfb3611cdf3d8da9eb7e571e9a8d74b02c7f8f94aa31d61012cdb3" gracePeriod=30
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:50.564188    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" containerID="cri-o://ef354a33f057b7b351508442e65d49eda49b50627ba9efb50cb1453ac27b6554" gracePeriod=30
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.652123    5078 generic.go:356] "Generic (PLEG): container finished" podID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerID="5a155a3b85f1461cac00c73db5137ff20b569a162b1005b5426884a75e779b2f" exitCode=0
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.652169    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" event={"ID":"86417f75-420a-4ef6-84ee-1d36bd8632ca","Type":"ContainerDied","Data":"5a155a3b85f1461cac00c73db5137ff20b569a162b1005b5426884a75e779b2f"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.652825    5078 scope.go:122] "RemoveContainer" containerID="5a155a3b85f1461cac00c73db5137ff20b569a162b1005b5426884a75e779b2f"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.655235    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.655328    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.655371    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.738806    5078 generic.go:356] "Generic (PLEG): container finished" podID="86073226-02be-4443-b886-654281e0a89a" containerID="e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec" exitCode=0
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.738848    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" event={"ID":"86073226-02be-4443-b886-654281e0a89a","Type":"ContainerDied","Data":"e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.739292    5078 scope.go:122] "RemoveContainer" containerID="e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.741062    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.741131    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:52.741164    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.752908    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" event={"ID":"0ac910e3-9861-4a2a-b293-e7ceb5df4e35","Type":"ContainerStarted","Data":"3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.752982    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.755133    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" event={"ID":"b5c1b29b-5502-46d7-bade-e05d3be82fe0","Type":"ContainerStarted","Data":"d496dc1bbb184391d6fdb60dcb462e62f6da06022b8fc2df0494343cca725fe7"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.755257    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.795155    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" podStartSLOduration=15.271738293 podStartE2EDuration="27.795138447s" totalImagesPullingTime="12.523400154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:53.779212269 +0000 UTC m=+1861.725582987" watchObservedRunningTime="2026-10-31 00:24:53.795138447 +0000 UTC m=+1861.741509164"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.819262    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" podStartSLOduration=17.323097032 podStartE2EDuration="29.819248472s" totalImagesPullingTime="12.49615144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:53.812329187 +0000 UTC m=+1861.758699922" watchObservedRunningTime="2026-10-31 00:24:53.819248472 +0000 UTC m=+1861.765619595"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.973068    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.973143    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.981926    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:53.983202    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:54.793188    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" event={"ID":"86073226-02be-4443-b886-654281e0a89a","Type":"ContainerStarted","Data":"c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:54.800584    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" event={"ID":"86417f75-420a-4ef6-84ee-1d36bd8632ca","Type":"ContainerStarted","Data":"9f126349b71a0ed63e287d00793de4ff76ead0d073f93537763560b26d45af8b"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:54.922057    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:54.984596    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:55.203747    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:55.251906    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:55.252018    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:55.252063    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:55.929085    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:55.929144    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.212981    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.213045    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.229579    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.229630    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.554985    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.555040    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.841214    5078 generic.go:356] "Generic (PLEG): container finished" podID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerID="ef354a33f057b7b351508442e65d49eda49b50627ba9efb50cb1453ac27b6554" exitCode=0
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.841280    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" event={"ID":"c7a113ee-02a0-456b-986c-1ad881dc630d","Type":"ContainerDied","Data":"ef354a33f057b7b351508442e65d49eda49b50627ba9efb50cb1453ac27b6554"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.843672    5078 generic.go:356] "Generic (PLEG): container finished" podID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerID="e7a4ea26dd6c40c4199c793c182c33b26766bbf6e67ee9f06e082a3d7d7db1be" exitCode=0
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.843732    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" event={"ID":"0c1d12b5-4ac9-46e9-a368-36f48a923c59","Type":"ContainerDied","Data":"e7a4ea26dd6c40c4199c793c182c33b26766bbf6e67ee9f06e082a3d7d7db1be"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.863830    5078 generic.go:356] "Generic (PLEG): container finished" podID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerID="00a6f8ab94bfb3611cdf3d8da9eb7e571e9a8d74b02c7f8f94aa31d61012cdb3" exitCode=0
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.863929    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" event={"ID":"1f1c1cec-fbce-4694-bb48-b6c32800f1e5","Type":"ContainerDied","Data":"00a6f8ab94bfb3611cdf3d8da9eb7e571e9a8d74b02c7f8f94aa31d61012cdb3"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.865819    5078 generic.go:356] "Generic (PLEG): container finished" podID="81265e6e-0c63-443e-9886-a53e9307af65" containerID="177dc1749989201340a07dbf9f171c0ba08ec26081e87ece26acc3a00831710b" exitCode=0
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.867205    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" event={"ID":"81265e6e-0c63-443e-9886-a53e9307af65","Type":"ContainerDied","Data":"177dc1749989201340a07dbf9f171c0ba08ec26081e87ece26acc3a00831710b"}
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.904829    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:56.904876    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:57.254381    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:57.254436    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:57.905337    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:57.905391    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.099686    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.123716    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.129639    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.143040    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.148922    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228223    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228300    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228345    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228387    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228455    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228511    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228550    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228578    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228635    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228667    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228706    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228751    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228808    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228847    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228896    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228931    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.228985    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229040    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229078    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229106    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229141    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229202    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229231    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229261    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229287    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229330    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229445    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229525    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229639    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca\") pod \"1563cf96-6d7c-4dc9-8911-710b77014179\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229692    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229738    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229817    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket\") pod \"81265e6e-0c63-443e-9886-a53e9307af65\" (UID: \"81265e6e-0c63-443e-9886-a53e9307af65\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229883    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.229969    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca\") pod \"1563cf96-6d7c-4dc9-8911-710b77014179\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.230062    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.230093    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9\" (UniqueName: \"kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9\") pod \"1563cf96-6d7c-4dc9-8911-710b77014179\" (UID: \"1563cf96-6d7c-4dc9-8911-710b77014179\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.230124    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.230179    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy\") pod \"c7a113ee-02a0-456b-986c-1ad881dc630d\" (UID: \"c7a113ee-02a0-456b-986c-1ad881dc630d\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.230234    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket\") pod \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\" (UID: \"1f1c1cec-fbce-4694-bb48-b6c32800f1e5\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.245321    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.251403    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.263685    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.263734    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.296650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca" pod "1563cf96-6d7c-4dc9-8911-710b77014179" (UID: "1563cf96-6d7c-4dc9-8911-710b77014179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.297271    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.297909    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.340166    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.340860    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.341819    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.342126    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.342663    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.343035    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.359188    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.360604    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.360741    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.365557    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.368561    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\") pod \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\" (UID: \"0c1d12b5-4ac9-46e9-a368-36f48a923c59\") "
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.370904    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.370930    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81265e6e-0c63-443e-9886-a53e9307af65-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.370946    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.370960    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.370974    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.370988    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371001    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371019    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371033    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371047    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371062    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371075    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:24:58.371161    5078 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0c1d12b5-4ac9-46e9-a368-36f48a923c59/volumes/kubernetes.io~empty-dir/workload-socket
Oct 31 00:24:59.034065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.371177    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.383440    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.398352    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.434642    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.435288    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.538736    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.538773    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.538792    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.538807    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.538821    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.540347    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.566634    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.566888    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.644808    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a113ee-02a0-456b-986c-1ad881dc630d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.644833    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.644842    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.739152    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "kube-api-access-v2l48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.739818    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca" pod "1563cf96-6d7c-4dc9-8911-710b77014179" (UID: "1563cf96-6d7c-4dc9-8911-710b77014179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.740899    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.752711    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "kube-api-access-2r2vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.753819    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.753921    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.754039    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.754112    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.754176    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token" pod "1f1c1cec-fbce-4694-bb48-b6c32800f1e5" (UID: "1f1c1cec-fbce-4694-bb48-b6c32800f1e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.754632    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9" pod "1563cf96-6d7c-4dc9-8911-710b77014179" (UID: "1563cf96-6d7c-4dc9-8911-710b77014179"). InnerVolumeSpecName "kube-api-access-c8kd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.755038    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758236    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758258    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758274    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2vn\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-kube-api-access-2r2vn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758288    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2l48\" (UniqueName: \"kubernetes.io/projected/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-kube-api-access-v2l48\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758303    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81265e6e-0c63-443e-9886-a53e9307af65-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758317    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c1cec-fbce-4694-bb48-b6c32800f1e5-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758330    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758343    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758357    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1563cf96-6d7c-4dc9-8911-710b77014179-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758371    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8kd9\" (UniqueName: \"kubernetes.io/projected/1563cf96-6d7c-4dc9-8911-710b77014179-kube-api-access-c8kd9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.758385    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.760569    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.766281    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.766326    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "kube-api-access-6tknn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.767977    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.767990    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy" pod "c7a113ee-02a0-456b-986c-1ad881dc630d" (UID: "c7a113ee-02a0-456b-986c-1ad881dc630d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.768203    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy" pod "81265e6e-0c63-443e-9886-a53e9307af65" (UID: "81265e6e-0c63-443e-9886-a53e9307af65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.768295    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm" pod "0c1d12b5-4ac9-46e9-a368-36f48a923c59" (UID: "0c1d12b5-4ac9-46e9-a368-36f48a923c59"). InnerVolumeSpecName "kube-api-access-mm7dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861591    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861654    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tknn\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-kube-api-access-6tknn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861673    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81265e6e-0c63-443e-9886-a53e9307af65-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861689    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81265e6e-0c63-443e-9886-a53e9307af65-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861703    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1d12b5-4ac9-46e9-a368-36f48a923c59-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861719    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a113ee-02a0-456b-986c-1ad881dc630d-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.861733    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm7dm\" (UniqueName: \"kubernetes.io/projected/0c1d12b5-4ac9-46e9-a368-36f48a923c59-kube-api-access-mm7dm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.927905    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:58.927968    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.042091    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.043971    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc" event={"ID":"c7a113ee-02a0-456b-986c-1ad881dc630d","Type":"ContainerDied","Data":"d1bb0848c206e7094b7c8738b4728a7975cf2e260ef36b967f4e1e9e210da8c0"}
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.044124    5078 scope.go:122] "RemoveContainer" containerID="ef354a33f057b7b351508442e65d49eda49b50627ba9efb50cb1453ac27b6554"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.047288    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr" event={"ID":"1563cf96-6d7c-4dc9-8911-710b77014179","Type":"ContainerDied","Data":"95b35762141381fb996293d94eb676ef9d28b36d3847480941aa4e4c32ac1859"}
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.047334    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b35762141381fb996293d94eb676ef9d28b36d3847480941aa4e4c32ac1859"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.047398    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.053368    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4" event={"ID":"0c1d12b5-4ac9-46e9-a368-36f48a923c59","Type":"ContainerDied","Data":"fdb6debcd3cb808b6d889261dc764216bd0bf52ddf5408edbcd8e0d19585479d"}
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.053506    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.055617    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.055649    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp" event={"ID":"1f1c1cec-fbce-4694-bb48-b6c32800f1e5","Type":"ContainerDied","Data":"cc1114b1032cc9cd7b050c561cd8fd1d492f907277af9348467fc0a696438105"}
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.086614    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8" event={"ID":"81265e6e-0c63-443e-9886-a53e9307af65","Type":"ContainerDied","Data":"3e9a34039312eb7db7c2bdd1ec18dcd4230b11af20cba8400a2db99712f29c4d"}
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.086735    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.251221    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.251279    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.351332    5078 scope.go:122] "RemoveContainer" containerID="e7a4ea26dd6c40c4199c793c182c33b26766bbf6e67ee9f06e082a3d7d7db1be"
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.405250    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"]
Oct 31 00:24:59.804093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.567487    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-blhc4"]
Oct 31 00:25:00.132618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.875263    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:25:00.132618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.969597    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:25:00.132618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.969660    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:25:00.132618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:24:59.970173    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:25:00.310767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.310101    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:25:00.310767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.310230    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:25:00.567887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.567752    5078 scope.go:122] "RemoveContainer" containerID="00a6f8ab94bfb3611cdf3d8da9eb7e571e9a8d74b02c7f8f94aa31d61012cdb3"
Oct 31 00:25:00.602315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.597228    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:00.851654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.750562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:25:00.851654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.820336    5078 scope.go:122] "RemoveContainer" containerID="177dc1749989201340a07dbf9f171c0ba08ec26081e87ece26acc3a00831710b"
Oct 31 00:25:00.851654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.822796    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:25:00.851654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.823139    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:00.875861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.875240    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:00.913972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.913890    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:00.915075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.915051    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:25:00.915259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.915235    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:25:00.933120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:00.933049    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:01.458644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.458603    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" path="/var/lib/kubelet/pods/0c1d12b5-4ac9-46e9-a368-36f48a923c59/volumes"
Oct 31 00:25:01.459958 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:25:01.459923    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.254s"
Oct 31 00:25:01.460271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.460256    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"]
Oct 31 00:25:01.460416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.460404    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-rk2cc"]
Oct 31 00:25:01.460795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.458989    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:25:01.461008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.460957    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:25:01.466832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.466311    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:01.490201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.489058    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:01.496327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.496290    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:01.539819 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.539734    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"]
Oct 31 00:25:01.820559 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.820518    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:01.821408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.821384    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:25:01.908022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.907989    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:25:01.908329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.908290    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:25:01.996475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:01.996418    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:02.060790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.058647    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:25:02.168879 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.168844    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-hc8x8"]
Oct 31 00:25:02.262513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.256032    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:25:02.354437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.354391    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:25:02.474550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.468675    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81265e6e-0c63-443e-9886-a53e9307af65" path="/var/lib/kubelet/pods/81265e6e-0c63-443e-9886-a53e9307af65/volumes"
Oct 31 00:25:02.500722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.488591    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" path="/var/lib/kubelet/pods/c7a113ee-02a0-456b-986c-1ad881dc630d/volumes"
Oct 31 00:25:02.500722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.489290    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"]
Oct 31 00:25:02.500722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.489328    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:25:02.500722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.489387    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:25:02.500722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.489420    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:25:02.500722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.497544    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-d5nrp"]
Oct 31 00:25:02.703493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.687087    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"]
Oct 31 00:25:02.833545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.813404    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1563cf96-6d7c-4dc9-8911-710b77014179" containerName="revoke-keys"
Oct 31 00:25:02.849517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.837324    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1563cf96-6d7c-4dc9-8911-710b77014179" containerName="revoke-keys"
Oct 31 00:25:02.900583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.900549    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy"
Oct 31 00:25:02.900812 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.900796    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy"
Oct 31 00:25:02.900898 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.900886    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy"
Oct 31 00:25:02.900963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.900954    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy"
Oct 31 00:25:02.901047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.901038    5078 cpu_manager.go:414] "containerMap: removing container" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy"
Oct 31 00:25:02.901112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.901104    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy"
Oct 31 00:25:02.901210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.901202    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy"
Oct 31 00:25:02.901270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.901262    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy"
Oct 31 00:25:02.927333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.927307    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1563cf96-6d7c-4dc9-8911-710b77014179" containerName="revoke-keys"
Oct 31 00:25:02.927605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.927592    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" containerName="istio-proxy"
Oct 31 00:25:03.200913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.200708    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1d12b5-4ac9-46e9-a368-36f48a923c59" containerName="istio-proxy"
Oct 31 00:25:03.201261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.201244    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="81265e6e-0c63-443e-9886-a53e9307af65" containerName="istio-proxy"
Oct 31 00:25:03.201408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.201396    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a113ee-02a0-456b-986c-1ad881dc630d" containerName="istio-proxy"
Oct 31 00:25:03.202678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:02.927836    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:25:03.202924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.202881    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:25:03.237440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.237403    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:25:03.237744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.237713    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:25:03.353654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.353616    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"]
Oct 31 00:25:03.354007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.353986    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.510811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.510708    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.511410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.511370    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.519643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.511589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.519643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.512266    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvjv\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.519643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.512652    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.519643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.512883    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.526898 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.526869    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.541649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.527224    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.541649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.527328    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.641509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.635018    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.641509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.635097    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.641509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.635152    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.641509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.635432    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.641509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.635985    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.641509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.636254    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.643780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.643737    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.643937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.643854    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.644004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.643948    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.644004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.643993    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvjv\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.644124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.644114    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.644216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.644196    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.662222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.653500    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.676903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.676493    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.771584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.766894    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:03.783737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.783675    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:03.801700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.801655    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:25:03.811841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.811767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.812672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.812646    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.849728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.839988    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:03.849728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.840837    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.887712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.878611    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:25:03.887712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.880943    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:25:03.949569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.940454    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvjv\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:03.953429 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.952994    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:25:03.953429 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.953069    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:25:03.962577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:03.962043    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.166484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.153339    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:04.166484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.153506    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.198622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.190499    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:04.237653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.237618    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:25:04.334704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.237847    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:25:04.334704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.258690    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.334704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.296006    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1c1cec-fbce-4694-bb48-b6c32800f1e5" path="/var/lib/kubelet/pods/1f1c1cec-fbce-4694-bb48-b6c32800f1e5/volumes"
Oct 31 00:25:04.334704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.305578    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:25:04.403275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.356938    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 31 00:25:04.473064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.470641    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.473258 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.473144    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 31 00:25:04.623665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.623596    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.702676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.702617    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 31 00:25:04.775583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.775489    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.806757 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.776125    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:25:04.808574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.808533    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:25:04.809283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.809261    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:25:04.812577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.812547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 31 00:25:04.830487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.814734    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:25:04.975980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.865581    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.975980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.927692    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.975980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.928113    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:04.975980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.929129    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:25:04.975980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.929193    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:25:04.975980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:04.967092    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:05.026639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.023699    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:25:05.079765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.073606    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 00:25:05.079765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.073693    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 00:25:05.293954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.293859    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 00:25:05.294602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.294546    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 00:25:05.316741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.316691    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:05.318257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.318233    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:25:05.345490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.330346    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Oct 31 00:25:05.355901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.355837    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:25:05.382999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.382968    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:25:05.571566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.560032    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:25:05.609979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.595067    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:05.869377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.756475    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:25:05.954486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.779562    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:05.954486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.827855    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:25:05.955686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.955648    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:25:05.955971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.955956    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:25:05.956701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.956683    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:25:05.956896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.956883    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:25:05.957034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:05.957024    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:25:06.122542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.122062    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:25:06.149620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.130979    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:25:06.416733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.391788    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:25:06.416733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.391860    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:25:06.416733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.391893    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:25:06.416733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.391942    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs"
Oct 31 00:25:06.445781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:06.445755    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:25:07.581877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:07.581829    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:07.582527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:07.582497    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:07.582722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:07.582702    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:07.887682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:07.885323    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"]
Oct 31 00:25:08.044135 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:25:08.044103    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:25:08.049059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:08.049029    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" event={"ID":"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7","Type":"ContainerStarted","Data":"65f27afad2a26a454e28e2a56a9d768cfeefe05f5ecb0d3ec6ae2c8d72490aff"}
Oct 31 00:25:08.646620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:08.644820    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"]
Oct 31 00:25:09.014499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.011592    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"]
Oct 31 00:25:09.014499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.011720    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.077207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.077175    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7b1fc7-openshift-default-dockercfg-qh6pn\""
Oct 31 00:25:09.133091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.132691    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.133091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.132737    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.133091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.132762    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.133091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.132966    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.200658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.133286    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnd86\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.200658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.133393    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.200658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.133445    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.200658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.133569    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.200658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.133624    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.245220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.245175    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.245603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.245577    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnd86\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.245897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.245871    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.246087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.246072    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.246334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.246310    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.246523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.246504    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.246803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.246786    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.246946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.246931    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.247079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.247064    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.266706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.254272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.266706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.255284    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.266706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.255560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.266706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.255794    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.266706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.260614    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.293488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.290125    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.312489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.302043    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" event={"ID":"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7","Type":"ContainerStarted","Data":"ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491"}
Oct 31 00:25:09.340628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.325237    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnd86\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.340628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.332853    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.340808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.340661    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy\") pod \"e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") " pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:09.482494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.474833    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podStartSLOduration=9.47481168 podStartE2EDuration="9.47481168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:09.371948739 +0000 UTC m=+1877.318319458" watchObservedRunningTime="2026-10-31 00:25:09.47481168 +0000 UTC m=+1877.421182398"
Oct 31 00:25:09.602939 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:09.602639    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:10.116598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.116563    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"]
Oct 31 00:25:10.194435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.193881    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 00:25:10.194435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.193972    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 00:25:10.523975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.505789    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"]
Oct 31 00:25:10.551053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.505906    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.551053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.524276    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"]
Oct 31 00:25:10.551053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.532171    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-34b786fc-openshift-default-dockercfg-7rgcz\""
Oct 31 00:25:10.642701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.642652    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.642892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.642725    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.642892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.642801    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbhh\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.642892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.642866    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.643048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.642944    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.643048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.642985    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.643161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.643138    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.643217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.643187    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.643567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.643456    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.714855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.652187    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"]
Oct 31 00:25:10.714855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.652238    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:10.714855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.652297    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.714855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.652395    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"]
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.744901    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.745060    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.745155    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.747733    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.747793    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.747834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.747961    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplqv\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.748088    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.748205    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.748333    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.748378    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.748832    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749077    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749082    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbhh\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749192    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749237    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749297    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749355    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749411    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749646    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749735    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749811    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749880    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.750108    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.753418    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.749650    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.754411    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:10.771011 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.755856    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.800453    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.822449    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbhh\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.852200    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplqv\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.852371    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.852670    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.852929    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.853059    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.853248    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.853287    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.853326    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.853372    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:10.995914    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.060922    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.061741    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.061866    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.061957    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.069401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.062062    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.131616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.108235    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.131616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.108782    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplqv\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.131616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.109223    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.131616 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.112451    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.202491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.193201    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 00:25:11.202491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.193285    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 00:25:11.601640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.516704    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:11.601640 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:25:10.732574    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a603b11_c6d9_4c14_932f_c1e3d1454bea.slice/crio-e89be8bbe624deb5853e41532befdd5e7ba5aaeaf5f0b2ec818655f4dc92077f WatchSource:0}: Error finding container e89be8bbe624deb5853e41532befdd5e7ba5aaeaf5f0b2ec818655f4dc92077f: Status 404 returned error can't find the container with id e89be8bbe624deb5853e41532befdd5e7ba5aaeaf5f0b2ec818655f4dc92077f
Oct 31 00:25:11.624343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.624263    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" event={"ID":"9a603b11-c6d9-4c14-932f-c1e3d1454bea","Type":"ContainerStarted","Data":"e89be8bbe624deb5853e41532befdd5e7ba5aaeaf5f0b2ec818655f4dc92077f"}
Oct 31 00:25:11.892591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:11.892558    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"]
Oct 31 00:25:12.055729 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:25:12.051822    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97548888_626e_4da5_b1c8_45b930ea1e72.slice/crio-dd31724fa4e2f0bb9b5d732c07e4097fc22ac7c126a5050009fd835bb0e9e2c9 WatchSource:0}: Error finding container dd31724fa4e2f0bb9b5d732c07e4097fc22ac7c126a5050009fd835bb0e9e2c9: Status 404 returned error can't find the container with id dd31724fa4e2f0bb9b5d732c07e4097fc22ac7c126a5050009fd835bb0e9e2c9
Oct 31 00:25:12.137246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.136246    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:12.137246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.136361    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:12.137246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.136408    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:12.232337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.232192    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 00:25:12.232337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.232248    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 00:25:12.643795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.643753    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"]
Oct 31 00:25:12.932281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.932217    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:12.938531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.938500    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:12.938754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.938739    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:12.942417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.942143    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" event={"ID":"97548888-626e-4da5-b1c8-45b930ea1e72","Type":"ContainerStarted","Data":"91c80b054825fda4518faf3f81549cc2db58d74b08260b5687b87553c1062e15"}
Oct 31 00:25:12.942417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.942212    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" event={"ID":"97548888-626e-4da5-b1c8-45b930ea1e72","Type":"ContainerStarted","Data":"dd31724fa4e2f0bb9b5d732c07e4097fc22ac7c126a5050009fd835bb0e9e2c9"}
Oct 31 00:25:12.944982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:12.944959    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" event={"ID":"9a603b11-c6d9-4c14-932f-c1e3d1454bea","Type":"ContainerStarted","Data":"4f1c2ec1c5698228c54c5c7e7827d66bed3da07fc9f105c2890f011b3b3a1b4e"}
Oct 31 00:25:13.035771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.035371    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podStartSLOduration=5.035349219 podStartE2EDuration="5.035349219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:12.994991469 +0000 UTC m=+1880.941362187" watchObservedRunningTime="2026-10-31 00:25:13.035349219 +0000 UTC m=+1880.981719937"
Oct 31 00:25:13.193223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.192878    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 00:25:13.193223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.192940    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 00:25:13.622531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.621917    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:13.633370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.632351    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:13.964245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.964197    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" event={"ID":"3db510f5-171b-43d9-9afc-6abc92243054","Type":"ContainerStarted","Data":"c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd"}
Oct 31 00:25:13.964447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.964275    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" event={"ID":"3db510f5-171b-43d9-9afc-6abc92243054","Type":"ContainerStarted","Data":"c80e1e7d586373c5ff98d2b3b68a1e5170e9fa926a8adf34259b1a0e314df028"}
Oct 31 00:25:13.965347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.965323    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:13.966428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.966411    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:25:13.997484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.997172    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:14.010530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.998548    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:14.010530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:13.998598    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:14.021820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.020678    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podStartSLOduration=4.020657595 podStartE2EDuration="4.020657595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:13.996782437 +0000 UTC m=+1881.943153146" watchObservedRunningTime="2026-10-31 00:25:14.020657595 +0000 UTC m=+1881.967028313"
Oct 31 00:25:14.041191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.041142    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podStartSLOduration=4.041127325 podStartE2EDuration="4.041127325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:14.040813797 +0000 UTC m=+1881.987184537" watchObservedRunningTime="2026-10-31 00:25:14.041127325 +0000 UTC m=+1881.987498082"
Oct 31 00:25:14.199534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.197968    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 00:25:14.199534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.198030    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 00:25:14.524380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.524342    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:14.524767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.524742    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:14.524956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:14.524916    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:15.088344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.067324    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:15.088344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.067385    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:15.214527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.214488    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 00:25:15.215156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.215126    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 00:25:15.534926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.534441    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:15.534926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.534529    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:16.033493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.999833    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:16.033493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:15.999895    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:16.467849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.467814    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:16.468603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.468558    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:16.558294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.472819    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:25:16.558294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.525844    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:16.558294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.525915    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:16.998856 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.998818    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:17.095629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:16.999026    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:17.525411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:17.524939    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:17.525411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:17.525014    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:17.920592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:17.913812    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:17.920592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:17.913867    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:18.024559 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:18.024513    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:18.024854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:18.024821    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:18.526488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:18.520154    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:18.526488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:18.520215    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:19.001632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:19.001596    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:19.009722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:19.001881    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:19.030498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:19.028825    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:19.030498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:19.028886    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:19.772490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:19.747703    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:19.772490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:19.747782    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:20.019489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:20.013709    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 00:25:20.019489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:20.013787    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 00:25:20.526499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:20.526017    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 00:25:20.526499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:20.526099    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 00:25:21.005626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:21.004625    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:21.051217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:21.006676    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:21.051217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:21.007630    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:25:21.906796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:21.906749    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:21.913511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:21.911140    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:21.913511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:21.913287    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:25:25.562058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:25.562024    5078 scope.go:122] "RemoveContainer" containerID="44abcaecb39656baa0433eba6e3a5e1777e94230e6bee169ba35528573294891"
Oct 31 00:25:25.723528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:25.723501    5078 scope.go:122] "RemoveContainer" containerID="b631f6e8ee0dc568404e57b5abbc9971f629af7c9ee0fd7ea2f0f6f8902124cf"
Oct 31 00:25:25.764611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:25.757110    5078 scope.go:122] "RemoveContainer" containerID="275609d3f15a6755e97b12baeff564cc97bf4c7680e254983699f1095356860a"
Oct 31 00:25:25.797134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:25.797109    5078 scope.go:122] "RemoveContainer" containerID="7d553752383a539f1c28d159dbfdc103cb83146244c102f83b92bbffa217e632"
Oct 31 00:25:25.885613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:25.885437    5078 scope.go:122] "RemoveContainer" containerID="39dcb51ffa769be4f7a9b627508c2bb3e4c50aeb1e1f76a0faabd78880d3ae03"
Oct 31 00:25:26.166810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.165698    5078 scope.go:122] "RemoveContainer" containerID="34746b0aaca066404969dd4a3688c181f16577e8b3ea9213886c00f87b98408d"
Oct 31 00:25:26.249117 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.247950    5078 scope.go:122] "RemoveContainer" containerID="3d2c9c87781a4bd0b854a0351b4494287221841182eca26ddb33ae04cdbfed1a"
Oct 31 00:25:26.293450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.293426    5078 scope.go:122] "RemoveContainer" containerID="9b1dc70e0def60076240d1055cfda405d06017608902f9b8031780102addfbe7"
Oct 31 00:25:26.334876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.325611    5078 scope.go:122] "RemoveContainer" containerID="7b9f2057ec0aebfea8a938a801494002f3a0f5087fadb39798288743531de0b2"
Oct 31 00:25:26.361023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.360982    5078 scope.go:122] "RemoveContainer" containerID="d8ac09d1e03aa3d9c57a2c8d467120b068d0b5e3e223f94521eebe400c67967e"
Oct 31 00:25:26.415046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.409891    5078 scope.go:122] "RemoveContainer" containerID="db69a05172307e96849f2753575e6c3e38796c2fa53100d65d6f861e4283d39d"
Oct 31 00:25:26.444283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.444257    5078 scope.go:122] "RemoveContainer" containerID="6634a064e89192fdf22d8d3b73866b99f86cf0c8702895d322ca01483a377445"
Oct 31 00:25:26.504244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.504209    5078 scope.go:122] "RemoveContainer" containerID="d88806fbf5ab4a9be8621919dd90b5dfacf139936887eb02e78c3167b2624d22"
Oct 31 00:25:26.783578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.781723    5078 scope.go:122] "RemoveContainer" containerID="944483f7a04a457707080c31f182ea000b8bc260e782a362c1dc70739bdfb073"
Oct 31 00:25:26.935417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.922187    5078 scope.go:122] "RemoveContainer" containerID="626ec469a2526fad061c53303b8e6c63b1ac5d8aeb4b4a36d00980e6f12c9f2a"
Oct 31 00:25:26.976021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:26.976000    5078 scope.go:122] "RemoveContainer" containerID="715907364b15dc74f4b24741a16ee080ca2da1326f021eccf9dee409973fd1e0"
Oct 31 00:25:32.871125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:32.870975    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:32.947994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:32.947965    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:32.948282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:32.948253    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:34.076929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:34.076661    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:34.076929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:34.076722    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:34.076929 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:34.076830    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:25:35.125033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:35.124996    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:35.125519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:35.125072    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:38.550455 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:25:38.550397    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:25:46.799724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:46.799630    5078 generic.go:356] "Generic (PLEG): container finished" podID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerID="9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c" exitCode=0
Oct 31 00:25:46.799724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:46.799690    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" event={"ID":"22f1a149-d6c7-4c5f-97c1-639866c3b6f5","Type":"ContainerDied","Data":"9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"}
Oct 31 00:25:46.822798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:46.800251    5078 scope.go:122] "RemoveContainer" containerID="9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"
Oct 31 00:25:46.907883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:46.907837    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:46.908080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:46.907920    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:46.908080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:46.907958    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:25:47.025233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.025200    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"]
Oct 31 00:25:47.065703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.065624    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.075979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.072561    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:25:47.075979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.072631    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:25:47.185597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.185560    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.185934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.185906    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.186070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.186054    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsss6\" (UniqueName: \"kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.288248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.288206    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsss6\" (UniqueName: \"kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.288532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.288513    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.288811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.288791    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.293978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.289205    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.465026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.464985    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.476013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.465100    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsss6\" (UniqueName: \"kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.725292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.722974    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:47.892490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.886255    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"]
Oct 31 00:25:47.928486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.915942    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:25:47.928486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.916005    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:25:47.929284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:47.929249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" event={"ID":"22f1a149-d6c7-4c5f-97c1-639866c3b6f5","Type":"ContainerStarted","Data":"593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083"}
Oct 31 00:25:48.124928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:48.124893    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:25:48.141346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:48.126067    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:48.141346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:48.126125    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:49.083496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:49.082834    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:25:49.124822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:49.124793    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:49.125050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:49.124877    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:50.125378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:50.125323    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:50.125925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:50.125408    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:51.125694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:51.125654    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:51.126180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:51.125734    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:51.618641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:51.618565    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"]
Oct 31 00:25:52.040291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:52.039966    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7" event={"ID":"0678b634-c15e-4e00-999e-7f4e87dd9fac","Type":"ContainerStarted","Data":"93b40a73be9f611c302bcc42ee09892d2274a0bcc3219b13b59e1ae035c6e64e"}
Oct 31 00:25:52.125977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:52.125404    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:52.125977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:52.125495    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:53.125390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:53.125354    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:53.125698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:53.125413    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:54.094587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:54.094547    5078 generic.go:356] "Generic (PLEG): container finished" podID="0678b634-c15e-4e00-999e-7f4e87dd9fac" containerID="88bd3643ec73affaa97ba32796139ecae97b8320469bbd592eb7930c79d3ac38" exitCode=0
Oct 31 00:25:54.095238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:54.095103    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7" event={"ID":"0678b634-c15e-4e00-999e-7f4e87dd9fac","Type":"ContainerDied","Data":"88bd3643ec73affaa97ba32796139ecae97b8320469bbd592eb7930c79d3ac38"}
Oct 31 00:25:54.097067 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:25:54.097037    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0678b634_c15e_4e00_999e_7f4e87dd9fac.slice/crio-conmon-88bd3643ec73affaa97ba32796139ecae97b8320469bbd592eb7930c79d3ac38.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:25:54.150533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:54.139849    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:54.150533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:54.139918    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:55.128304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:55.127189    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 00:25:55.128304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:55.127279    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 00:25:56.135864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.135837    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7" event={"ID":"0678b634-c15e-4e00-999e-7f4e87dd9fac","Type":"ContainerDied","Data":"93b40a73be9f611c302bcc42ee09892d2274a0bcc3219b13b59e1ae035c6e64e"}
Oct 31 00:25:56.136323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.135889    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b40a73be9f611c302bcc42ee09892d2274a0bcc3219b13b59e1ae035c6e64e"
Oct 31 00:25:56.151109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.151083    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:25:56.152416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.152396    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:25:56.162678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.162652    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:25:56.229140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.229067    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6\" (UniqueName: \"kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6\") pod \"0678b634-c15e-4e00-999e-7f4e87dd9fac\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") "
Oct 31 00:25:56.229418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.229386    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca\") pod \"0678b634-c15e-4e00-999e-7f4e87dd9fac\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") "
Oct 31 00:25:56.229608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.229523    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca\") pod \"0678b634-c15e-4e00-999e-7f4e87dd9fac\" (UID: \"0678b634-c15e-4e00-999e-7f4e87dd9fac\") "
Oct 31 00:25:56.229875 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.229845    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca" pod "0678b634-c15e-4e00-999e-7f4e87dd9fac" (UID: "0678b634-c15e-4e00-999e-7f4e87dd9fac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:25:56.230876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.230854    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:56.296175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.296136    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca" pod "0678b634-c15e-4e00-999e-7f4e87dd9fac" (UID: "0678b634-c15e-4e00-999e-7f4e87dd9fac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:25:56.296853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.296649    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6" pod "0678b634-c15e-4e00-999e-7f4e87dd9fac" (UID: "0678b634-c15e-4e00-999e-7f4e87dd9fac"). InnerVolumeSpecName "kube-api-access-bsss6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:25:56.353848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.353708    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsss6\" (UniqueName: \"kubernetes.io/projected/0678b634-c15e-4e00-999e-7f4e87dd9fac-kube-api-access-bsss6\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:56.353848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:56.353738    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0678b634-c15e-4e00-999e-7f4e87dd9fac-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:57.157225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:25:57.157169    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"
Oct 31 00:26:02.909533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:02.908816    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:26:02.909533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:02.908883    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:02.909533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:02.909033    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:26:03.030315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:03.030261    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"]
Oct 31 00:26:03.040644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:03.039651    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21935661-2fzc7"]
Oct 31 00:26:03.057754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:03.057721    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"]
Oct 31 00:26:03.086206 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:03.085634    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"]
Oct 31 00:26:03.955637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:03.955603    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:26:03.956029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:03.955665    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:04.234726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:04.234690    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26289741-7ab5-4810-8edf-dd77dfb9663d" path="/var/lib/kubelet/pods/26289741-7ab5-4810-8edf-dd77dfb9663d/volumes"
Oct 31 00:26:05.560721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.560685    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"]
Oct 31 00:26:05.561713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.561690    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0678b634-c15e-4e00-999e-7f4e87dd9fac" containerName="revoke-keys"
Oct 31 00:26:05.561713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.561712    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0678b634-c15e-4e00-999e-7f4e87dd9fac" containerName="revoke-keys"
Oct 31 00:26:05.561987 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.561969    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0678b634-c15e-4e00-999e-7f4e87dd9fac" containerName="revoke-keys"
Oct 31 00:26:05.752697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.752658    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"]
Oct 31 00:26:05.752883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.752788    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.759769 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.759746    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:26:05.760023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.760008    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:26:05.859813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.859729    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.926225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.859870    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.926225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.860095    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxkd\" (UniqueName: \"kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.977751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.963252    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.977751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.963393    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.977751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.963810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxkd\" (UniqueName: \"kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.977751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.965685    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.977751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.976831    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:05.994251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:05.994215    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxkd\" (UniqueName: \"kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd\") pod \"maas-api-revoke-sub-5d9de170-pqhd4\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:06.239598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:06.239561    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:06.725271 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:06.546105    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:26:07.266752 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:26:07.266712    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode986cf3c_6bc4_4a9e_9777_e51e1f45c2b6.slice/crio-43cef53515c4be98c9899b1da4d78a53259eacb0796a7cec0806be3f38da9f45 WatchSource:0}: Error finding container 43cef53515c4be98c9899b1da4d78a53259eacb0796a7cec0806be3f38da9f45: Status 404 returned error can't find the container with id 43cef53515c4be98c9899b1da4d78a53259eacb0796a7cec0806be3f38da9f45
Oct 31 00:26:07.267791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:07.267769    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"]
Oct 31 00:26:07.290373 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:07.290343    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4" event={"ID":"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6","Type":"ContainerStarted","Data":"43cef53515c4be98c9899b1da4d78a53259eacb0796a7cec0806be3f38da9f45"}
Oct 31 00:26:08.112899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:08.107545    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing" containerID="cri-o://2b01d87a1755dcf35835c595c30550a6872ae58269423156f68be84f30801498" gracePeriod=25
Oct 31 00:26:08.117101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:08.114398    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing" containerID="cri-o://2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3" gracePeriod=25
Oct 31 00:26:09.314412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:09.314364    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4" event={"ID":"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6","Type":"ContainerStarted","Data":"11ab478d9195f5d55c810081d1ee7ddd493b8a33256b10de14ba390e7aac5cc5"}
Oct 31 00:26:09.344355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:09.344309    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4" podStartSLOduration=4.344295377 podStartE2EDuration="4.344295377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:09.341244437 +0000 UTC m=+1937.287615155" watchObservedRunningTime="2026-10-31 00:26:09.344295377 +0000 UTC m=+1937.290666093"
Oct 31 00:26:11.337652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:11.337618    5078 generic.go:356] "Generic (PLEG): container finished" podID="e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" containerID="11ab478d9195f5d55c810081d1ee7ddd493b8a33256b10de14ba390e7aac5cc5" exitCode=0
Oct 31 00:26:11.338161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:11.337690    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4" event={"ID":"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6","Type":"ContainerDied","Data":"11ab478d9195f5d55c810081d1ee7ddd493b8a33256b10de14ba390e7aac5cc5"}
Oct 31 00:26:12.824144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.823126    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:26:12.924395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.825168    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:26:12.924395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.916975    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:12.947849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.947820    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca\") pod \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") "
Oct 31 00:26:12.948028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.947867    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca\") pod \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") "
Oct 31 00:26:12.948264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.948245    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd\" (UniqueName: \"kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd\") pod \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\" (UID: \"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6\") "
Oct 31 00:26:12.948264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.948248    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca" pod "e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" (UID: "e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:12.949309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.949280    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:12.950335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.950312    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca" pod "e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" (UID: "e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:26:12.950592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:12.950573    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd" pod "e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" (UID: "e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6"). InnerVolumeSpecName "kube-api-access-4fxkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:13.051586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:13.051516    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxkd\" (UniqueName: \"kubernetes.io/projected/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-kube-api-access-4fxkd\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:13.051586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:13.051557    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:13.605418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:13.605380    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4" event={"ID":"e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6","Type":"ContainerDied","Data":"43cef53515c4be98c9899b1da4d78a53259eacb0796a7cec0806be3f38da9f45"}
Oct 31 00:26:13.605676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:13.605447    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43cef53515c4be98c9899b1da4d78a53259eacb0796a7cec0806be3f38da9f45"
Oct 31 00:26:13.605676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:13.605542    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"
Oct 31 00:26:17.715279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:17.715242    5078 generic.go:356] "Generic (PLEG): container finished" podID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerID="19cb826b1ec78cdb5567f5c750c2b4a34271a20f8325ad8baab9558b3bbc6714" exitCode=0
Oct 31 00:26:17.715794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:17.715298    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" event={"ID":"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7","Type":"ContainerDied","Data":"19cb826b1ec78cdb5567f5c750c2b4a34271a20f8325ad8baab9558b3bbc6714"}
Oct 31 00:26:17.715916 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:17.715900    5078 scope.go:122] "RemoveContainer" containerID="19cb826b1ec78cdb5567f5c750c2b4a34271a20f8325ad8baab9558b3bbc6714"
Oct 31 00:26:17.718004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:17.717975    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:17.718105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:17.718050    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:17.718105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:17.718082    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:18.739228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:18.739187    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" event={"ID":"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7","Type":"ContainerStarted","Data":"2a86a0c197d966969edd953a281bb3f52a16de6b30bfc2b2305cf010b85bd695"}
Oct 31 00:26:18.907372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:18.907341    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"]
Oct 31 00:26:18.925241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:18.907569    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy" containerID="cri-o://bc55c8c35ae3edb7c71643b0df613a0601f44d9e81e174ec5553e58e5d6354db" gracePeriod=30
Oct 31 00:26:18.951789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:18.947653    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:26:18.951789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:18.948337    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:26:18.951789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:18.948396    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:26:19.947712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:19.947681    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:26:19.948128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:19.947727    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:26:21.006242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:21.006207    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:26:21.047771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:21.047728    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:26:21.947477 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:21.947428    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:26:21.948032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:21.947540    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:26:22.823584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:22.823541    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:26:22.825181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:22.825156    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:26:22.947450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:22.947414    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:26:22.947450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:22.947484    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:26:23.947577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:23.947074    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 31 00:26:23.947577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:23.947152    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 31 00:26:24.128282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:24.128242    5078 generic.go:356] "Generic (PLEG): container finished" podID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerID="bc55c8c35ae3edb7c71643b0df613a0601f44d9e81e174ec5553e58e5d6354db" exitCode=0
Oct 31 00:26:24.215585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:24.128285    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" event={"ID":"865f19cd-c107-4a5c-8ec6-64566ede3bf4","Type":"ContainerDied","Data":"bc55c8c35ae3edb7c71643b0df613a0601f44d9e81e174ec5553e58e5d6354db"}
Oct 31 00:26:24.215585 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:24.131264    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865f19cd_c107_4a5c_8ec6_64566ede3bf4.slice/crio-bc55c8c35ae3edb7c71643b0df613a0601f44d9e81e174ec5553e58e5d6354db.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:24.954671    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:24.956136    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:24.956969    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.115372    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.142069    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.142106    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t" event={"ID":"865f19cd-c107-4a5c-8ec6-64566ede3bf4","Type":"ContainerDied","Data":"f9abb3a4e6d531128c089c672f82a6a74ea8107d5f6c919c2d7b9cfb86e776ec"}
Oct 31 00:26:25.180839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.142190    5078 scope.go:122] "RemoveContainer" containerID="bc55c8c35ae3edb7c71643b0df613a0601f44d9e81e174ec5553e58e5d6354db"
Oct 31 00:26:25.245281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.245237    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.245439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.245308    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.245439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.245350    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.245439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.245400    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.245531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246186    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.245822    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246240    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246334    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246372    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246424    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246450    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246505    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb\") pod \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\" (UID: \"865f19cd-c107-4a5c-8ec6-64566ede3bf4\") "
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.246740    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.247573    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.247594    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.247610    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.247623    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.247635    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.273889    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"]
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.275117    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" containerName="revoke-keys"
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.275136    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" containerName="revoke-keys"
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.275162    5078 cpu_manager.go:414] "containerMap: removing container" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy"
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.275169    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy"
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.275421    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" containerName="revoke-keys"
Oct 31 00:26:25.289780 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.275437    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" containerName="istio-proxy"
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.328758    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"]
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.328876    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.330959    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "kube-api-access-69phb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.331046    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.337234    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.337830    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token" pod "865f19cd-c107-4a5c-8ec6-64566ede3bf4" (UID: "865f19cd-c107-4a5c-8ec6-64566ede3bf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.359618    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69phb\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-kube-api-access-69phb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.359636    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.359646    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.375254 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.359655    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865f19cd-c107-4a5c-8ec6-64566ede3bf4-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:25.549646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.549551    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.549646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.549618    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.549686    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.549795    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.550037    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.550069    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.550096    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jhn\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.550134    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.550910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.550195    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.628795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.628755    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"]
Oct 31 00:26:25.652592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.652559    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.652756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.652674    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.652933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.652910    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.652944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.652969    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jhn\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653000    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653049    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653163    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653263    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653273    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653300    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.653551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653394    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.668069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653692    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.668069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.653734    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.668069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.655712    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-wfp5t"]
Oct 31 00:26:25.668069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.663279    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.668069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.663628    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.675902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.675884    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jhn\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.683165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.683146    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-7787w\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:25.908491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:25.905863    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:26.199186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:26.199107    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865f19cd-c107-4a5c-8ec6-64566ede3bf4" path="/var/lib/kubelet/pods/865f19cd-c107-4a5c-8ec6-64566ede3bf4/volumes"
Oct 31 00:26:26.278896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:26.265295    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"]
Oct 31 00:26:26.278896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:26.265878    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:26.278896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:26.265945    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:26.278896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:26.265973    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:27.191193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:27.191157    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" event={"ID":"790147f1-a37a-4923-9375-9776c7635605","Type":"ContainerStarted","Data":"220dc17a9beadfbc1b81321b4b577cc377de699f94ea0e91f86f2609cc67a5f2"}
Oct 31 00:26:27.191562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:27.191211    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" event={"ID":"790147f1-a37a-4923-9375-9776c7635605","Type":"ContainerStarted","Data":"e1474f2a9828e529b19089e88177d41013766be940a6cc7c4769a909f90a817b"}
Oct 31 00:26:27.231996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:27.231923    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podStartSLOduration=40.23190343 podStartE2EDuration="40.23190343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:27.22284866 +0000 UTC m=+1955.169219406" watchObservedRunningTime="2026-10-31 00:26:27.23190343 +0000 UTC m=+1955.178274148"
Oct 31 00:26:27.907947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:27.907604    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:27.909200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:27.909172    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-7787w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 00:26:27.909399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:27.909377    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 00:26:28.926209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:28.926148    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-7787w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 00:26:28.927036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:28.926225    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 00:26:29.016088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.013578    5078 scope.go:122] "RemoveContainer" containerID="50530c2eb575c290e8dc190065789d29fbf2a29e7765a0efc0ee943fd9e036a0"
Oct 31 00:26:29.130663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.130639    5078 scope.go:122] "RemoveContainer" containerID="0d50cc426fd6b1287ea7d8e44bcda5a7d4a2154b60ed71e0b56c4000a4c551de"
Oct 31 00:26:29.232027 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.231986    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:26:29.239977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.239944    5078 generic.go:356] "Generic (PLEG): container finished" podID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerID="2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3" exitCode=0
Oct 31 00:26:29.240232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.240212    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" event={"ID":"1e43b897-6637-4e7d-8faa-0650c2d1cfef","Type":"ContainerDied","Data":"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"}
Oct 31 00:26:29.240321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.240268    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc" event={"ID":"1e43b897-6637-4e7d-8faa-0650c2d1cfef","Type":"ContainerDied","Data":"3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616"}
Oct 31 00:26:29.240321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.240303    5078 scope.go:122] "RemoveContainer" containerID="2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"
Oct 31 00:26:29.240450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.240433    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"
Oct 31 00:26:29.243161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.243138    5078 generic.go:356] "Generic (PLEG): container finished" podID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerID="2b01d87a1755dcf35835c595c30550a6872ae58269423156f68be84f30801498" exitCode=0
Oct 31 00:26:29.244513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.244490    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" event={"ID":"08725c5f-43e2-4fb4-8cb6-fd0549615d07","Type":"ContainerDied","Data":"2b01d87a1755dcf35835c595c30550a6872ae58269423156f68be84f30801498"}
Oct 31 00:26:29.244693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.244673    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n" event={"ID":"08725c5f-43e2-4fb4-8cb6-fd0549615d07","Type":"ContainerDied","Data":"f2e3fff57c82d904c22b0b03bc8844d4c4e9ea60793955550a169b01e8e62d96"}
Oct 31 00:26:29.244809 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.244795    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e3fff57c82d904c22b0b03bc8844d4c4e9ea60793955550a169b01e8e62d96"
Oct 31 00:26:29.246664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.246649    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:26:29.256708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.256686    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume\") pod \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") "
Oct 31 00:26:29.256838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.256799    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj\" (UniqueName: \"kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj\") pod \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\" (UID: \"1e43b897-6637-4e7d-8faa-0650c2d1cfef\") "
Oct 31 00:26:29.257276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.257255    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume" pod "1e43b897-6637-4e7d-8faa-0650c2d1cfef" (UID: "1e43b897-6637-4e7d-8faa-0650c2d1cfef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:29.257909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.257879    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43b897-6637-4e7d-8faa-0650c2d1cfef-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:29.267968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.267936    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj" pod "1e43b897-6637-4e7d-8faa-0650c2d1cfef" (UID: "1e43b897-6637-4e7d-8faa-0650c2d1cfef"). InnerVolumeSpecName "kube-api-access-q62lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:29.334964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.333918    5078 scope.go:122] "RemoveContainer" containerID="2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"
Oct 31 00:26:29.343990 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.343969    5078 scope.go:122] "RemoveContainer" containerID="af06753612df4bf0f25ab7b83acea607389d8b8385e011815dfcc61b9004e05b"
Oct 31 00:26:29.359188 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.359146    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"]
Oct 31 00:26:29.360708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.360668    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing"
Oct 31 00:26:29.361076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.361058    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing"
Oct 31 00:26:29.361391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.361374    5078 cpu_manager.go:414] "containerMap: removing container" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing"
Oct 31 00:26:29.361626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.361612    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing"
Oct 31 00:26:29.362021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.361996    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" containerName="payload-pre-processing"
Oct 31 00:26:29.362153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.362142    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" containerName="payload-processing"
Oct 31 00:26:29.365290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.361182    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume" pod "08725c5f-43e2-4fb4-8cb6-fd0549615d07" (UID: "08725c5f-43e2-4fb4-8cb6-fd0549615d07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:29.469729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.360819    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume\") pod \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") "
Oct 31 00:26:29.469729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.373812    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl\" (UniqueName: \"kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl\") pod \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\" (UID: \"08725c5f-43e2-4fb4-8cb6-fd0549615d07\") "
Oct 31 00:26:29.469729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.375964    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725c5f-43e2-4fb4-8cb6-fd0549615d07-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:29.469729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.375993    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62lj\" (UniqueName: \"kubernetes.io/projected/1e43b897-6637-4e7d-8faa-0650c2d1cfef-kube-api-access-q62lj\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:29.559888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.555977    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl" pod "08725c5f-43e2-4fb4-8cb6-fd0549615d07" (UID: "08725c5f-43e2-4fb4-8cb6-fd0549615d07"). InnerVolumeSpecName "kube-api-access-r6zzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:29.716524 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:29.699530    5078 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc_openshift-ingress_1e43b897-6637-4e7d-8faa-0650c2d1cfef_0 in pod sandbox 3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616 from index: no such id: '2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3'" containerID="2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"
Oct 31 00:26:29.716524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.699617    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"} err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc_openshift-ingress_1e43b897-6637-4e7d-8faa-0650c2d1cfef_0 in pod sandbox 3643585e029b6b1dd7f8a6c3aac80e480ef452867c9e953dc30b454a9a7c2616 from index: no such id: '2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3'"
Oct 31 00:26:29.716524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:29.699654    5078 scope.go:122] "RemoveContainer" containerID="2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"
Oct 31 00:26:30.135636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.113358    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"]
Oct 31 00:26:30.138619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.136773    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.142331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.138362    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zzl\" (UniqueName: \"kubernetes.io/projected/08725c5f-43e2-4fb4-8cb6-fd0549615d07-kube-api-access-r6zzl\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:30.142961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.142932    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-7787w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 00:26:30.144007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.143969    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 00:26:30.153931 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:30.153896    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3\": container with ID starting with 2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3 not found: ID does not exist" containerID="2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"
Oct 31 00:26:30.154163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.154113    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3"} err="failed to get container status \"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3\": rpc error: code = NotFound desc = could not find container \"2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3\": container with ID starting with 2372e4039b17283094b1b2b9b319ffa0fc0169f01a4306cf68d30de788bc54a3 not found: ID does not exist"
Oct 31 00:26:30.168375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.168353    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7b1fc7-dockercfg-66wkm\""
Oct 31 00:26:30.180453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.168593    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7b1fc7\""
Oct 31 00:26:30.256302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.256282    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"
Oct 31 00:26:30.321574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.321545    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"]
Oct 31 00:26:30.325798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.325777    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-19adb9-6bf79b5757-krpsc"]
Oct 31 00:26:30.346119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.346081    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lw7\" (UniqueName: \"kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7\") pod \"payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.346502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.346444    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.408410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.408329    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"]
Oct 31 00:26:30.415656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.415302    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-19adb9-7d79f6bb5f-5k65n"]
Oct 31 00:26:30.448010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.447972    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.448279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.448259    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lw7\" (UniqueName: \"kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7\") pod \"payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.448756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.448734    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.464868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.464839    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lw7\" (UniqueName: \"kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7\") pod \"payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.547055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.547021    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:30.908869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.908301    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-7787w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 00:26:30.908869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:30.908356    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 00:26:31.098536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:31.087780    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"]
Oct 31 00:26:31.539489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:31.530010    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm" event={"ID":"cabd2b76-1571-4b37-85bc-5588b382f10f","Type":"ContainerStarted","Data":"55d5e0214b40ce054f6aa2dd3e86fd48c7cd4e4bb6a867fea781fefb955e7037"}
Oct 31 00:26:31.910923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:31.910808    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-7787w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 00:26:31.910923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:31.910861    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 00:26:32.297531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.297410    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08725c5f-43e2-4fb4-8cb6-fd0549615d07" path="/var/lib/kubelet/pods/08725c5f-43e2-4fb4-8cb6-fd0549615d07/volumes"
Oct 31 00:26:32.336853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.336816    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e43b897-6637-4e7d-8faa-0650c2d1cfef" path="/var/lib/kubelet/pods/1e43b897-6637-4e7d-8faa-0650c2d1cfef/volumes"
Oct 31 00:26:32.598751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.597204    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm" event={"ID":"cabd2b76-1571-4b37-85bc-5588b382f10f","Type":"ContainerStarted","Data":"003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254"}
Oct 31 00:26:32.598751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.598636    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:32.626574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.626513    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm" podStartSLOduration=72.626493804 podStartE2EDuration="1m12.626493804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:32.624886745 +0000 UTC m=+1960.571257463" watchObservedRunningTime="2026-10-31 00:26:32.626493804 +0000 UTC m=+1960.572864523"
Oct 31 00:26:32.918505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.917947    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-7787w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 00:26:32.918505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:32.918073    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 00:26:33.941935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:33.941891    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:33.942762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:33.942736    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:33.948009 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:33.944498    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:26:36.410487 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:36.409717    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:26:40.116628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.116586    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"]
Oct 31 00:26:40.122019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.121990    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-wnhr7"]
Oct 31 00:26:40.254758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.254723    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0678b634-c15e-4e00-999e-7f4e87dd9fac" path="/var/lib/kubelet/pods/0678b634-c15e-4e00-999e-7f4e87dd9fac/volumes"
Oct 31 00:26:40.255122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.255103    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"]
Oct 31 00:26:40.255218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.255167    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"]
Oct 31 00:26:40.255869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.255444    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" containerID="cri-o://ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2" gracePeriod=30
Oct 31 00:26:40.257159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:40.257118    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" containerID="cri-o://e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84" gracePeriod=30
Oct 31 00:26:41.586734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:41.586697    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:26:41.587225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:41.586780    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:26:43.165505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:43.165456    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:26:43.165915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:43.165525    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:26:44.634016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:44.633988    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:26:45.541041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.540959    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:26:45.621548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.621515    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"]
Oct 31 00:26:45.626314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.626285    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy"
Oct 31 00:26:45.626314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.626310    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy"
Oct 31 00:26:45.626604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.626324    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy"
Oct 31 00:26:45.626604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.626329    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy"
Oct 31 00:26:45.626604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.626558    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy"
Oct 31 00:26:45.626604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.626570    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3aadd02-d322-476b-893b-2459b5f12808" containerName="istio-proxy"
Oct 31 00:26:45.634192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.634161    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.644786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.644760    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"]
Oct 31 00:26:45.733323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733278    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.733542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733381    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.733542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733476    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.733682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733598    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.733738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733689    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.733794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733757    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.733847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733800    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.734725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733869    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.734725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.733979    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8\") pod \"f3aadd02-d322-476b-893b-2459b5f12808\" (UID: \"f3aadd02-d322-476b-893b-2459b5f12808\") "
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.746680    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.746857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.746894    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.747006    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxfn\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.747071    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.747099    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.748166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.747227    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.788770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.750405    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:45.788770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.751025    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:45.788770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.751341    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:45.788770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.751601    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:45.788770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.751976    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:45.788770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.781163    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:45.798328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798240    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.798611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798434    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.798709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798652    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aadd02-d322-476b-893b-2459b5f12808-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.798709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798672    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.798709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798690    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.798709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798705    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.798921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798719    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.798921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.798733    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aadd02-d322-476b-893b-2459b5f12808-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.806247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.806216    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:45.806567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.806543    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "kube-api-access-d7wj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:45.806884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.806858    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo" pod "f3aadd02-d322-476b-893b-2459b5f12808" (UID: "f3aadd02-d322-476b-893b-2459b5f12808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:26:45.884347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.872701    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900035    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900169    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900245    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900394    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900422    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900573    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxfn\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900619    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900641    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900731    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900926    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aadd02-d322-476b-893b-2459b5f12808-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900945    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wj8\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-kube-api-access-d7wj8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.900963    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aadd02-d322-476b-893b-2459b5f12808-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.901599    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.901840    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.902582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.902414    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.903947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.903811    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.906615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.904292    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.923862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.923836    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxfn\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.948081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.948001    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.957038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.955699    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"]
Oct 31 00:26:45.957481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.957404    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy"
Oct 31 00:26:45.957481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.957426    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy"
Oct 31 00:26:45.958489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.957935    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerName="istio-proxy"
Oct 31 00:26:45.963132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.963102    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:45.966487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.964809    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:45.994540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:45.994511    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:46.008218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.002801    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.008412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.007881    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"]
Oct 31 00:26:46.009609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.009167    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.009609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.009277    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.009609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.009306    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.009609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.009402    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.010261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.010049    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.010942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.010389    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.010942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.010622    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.010942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.010698    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo\") pod \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\" (UID: \"c916afdf-77b9-4ffc-8c9e-3b89f67269b8\") "
Oct 31 00:26:46.011885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.011554    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:46.012966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.012154    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.012966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.012663    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "kube-api-access-h5nqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:46.013127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.013003    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:46.016640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.013268    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:46.016640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.013352    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:46.016640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.015302    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:46.018304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.017455    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:26:46.019427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.019406    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:46.021005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.020970    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy" pod "c916afdf-77b9-4ffc-8c9e-3b89f67269b8" (UID: "c916afdf-77b9-4ffc-8c9e-3b89f67269b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:46.113707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.113620    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.113707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.113674    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.113910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.113885    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114048    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114097    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114313    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114405    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqm8z\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114505    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114597    5078 generic.go:356] "Generic (PLEG): container finished" podID="f3aadd02-d322-476b-893b-2459b5f12808" containerID="ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2" exitCode=0
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114605    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114681    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114694    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" event={"ID":"f3aadd02-d322-476b-893b-2459b5f12808","Type":"ContainerDied","Data":"ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2"}
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114755    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7" event={"ID":"f3aadd02-d322-476b-893b-2459b5f12808","Type":"ContainerDied","Data":"0cc2068a8ef146ab0d3f454f808174c19ec117c8352c006923d48ff073d09fd7"}
Oct 31 00:26:46.114899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114776    5078 scope.go:122] "RemoveContainer" containerID="ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2"
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.114970    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115182    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115215    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115230    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115246    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115260    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115272    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5nqj\" (UniqueName: \"kubernetes.io/projected/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-kube-api-access-h5nqj\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.115499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.115286    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c916afdf-77b9-4ffc-8c9e-3b89f67269b8-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:46.116874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.116826    5078 generic.go:356] "Generic (PLEG): container finished" podID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" containerID="e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84" exitCode=0
Oct 31 00:26:46.117015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.116923    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"
Oct 31 00:26:46.117015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.116942    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" event={"ID":"c916afdf-77b9-4ffc-8c9e-3b89f67269b8","Type":"ContainerDied","Data":"e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84"}
Oct 31 00:26:46.117015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.116984    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26" event={"ID":"c916afdf-77b9-4ffc-8c9e-3b89f67269b8","Type":"ContainerDied","Data":"062293a84fd2c7da09ccf41d3c4aa2173733be8c4b08de5046fb61c0a4eb9caa"}
Oct 31 00:26:46.138145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.138126    5078 scope.go:122] "RemoveContainer" containerID="f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.151345    5078 scope.go:122] "RemoveContainer" containerID="ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:46.151708    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2\": container with ID starting with ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2 not found: ID does not exist" containerID="ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.151780    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2"} err="failed to get container status \"ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2\": rpc error: code = NotFound desc = could not find container \"ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2\": container with ID starting with ef634260637f7ba61a9a22199f6da417f11c3a6964a9dcede26645b39f5549a2 not found: ID does not exist"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.151799    5078 scope.go:122] "RemoveContainer" containerID="f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:46.152052    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755\": container with ID starting with f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755 not found: ID does not exist" containerID="f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.152074    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755"} err="failed to get container status \"f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755\": rpc error: code = NotFound desc = could not find container \"f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755\": container with ID starting with f4d0fca87b6b1a44617de46a74e61a77e0b2fdf8b72eb772d673de09a259e755 not found: ID does not exist"
Oct 31 00:26:46.152389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.152088    5078 scope.go:122] "RemoveContainer" containerID="e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84"
Oct 31 00:26:46.158534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.158512    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"]
Oct 31 00:26:46.169581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.169552    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-6jb26"]
Oct 31 00:26:46.178474 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.178442    5078 scope.go:122] "RemoveContainer" containerID="e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84"
Oct 31 00:26:46.179072 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:46.178930    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84\": container with ID starting with e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84 not found: ID does not exist" containerID="e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84"
Oct 31 00:26:46.179072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.178986    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84"} err="failed to get container status \"e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84\": rpc error: code = NotFound desc = could not find container \"e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84\": container with ID starting with e11bd5265afa1ca142d3723db33955499555c86ec291ecdc4117bb24cb746a84 not found: ID does not exist"
Oct 31 00:26:46.183876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.183851    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"]
Oct 31 00:26:46.203127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.203099    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c916afdf-77b9-4ffc-8c9e-3b89f67269b8" path="/var/lib/kubelet/pods/c916afdf-77b9-4ffc-8c9e-3b89f67269b8/volumes"
Oct 31 00:26:46.203659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.203638    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-g9wn7"]
Oct 31 00:26:46.217161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217141    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217225    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217302    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217326    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217623    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217699    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqm8z\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217758    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217809    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.217874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217816    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.218309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.217920    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.218309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.218106    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.218309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.218159    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.218309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.218219    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.218542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.218363    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.220611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.220591    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.221123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.221105    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.231924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.231898    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.232033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.232015    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqm8z\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.260139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.260115    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:46.333997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.333823    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:46.576647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.576621    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"]
Oct 31 00:26:46.588119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.588089    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:46.588297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.588280    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:46.588422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.588404    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:46.685631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.685475    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"]
Oct 31 00:26:46.686122 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:26:46.686091    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b949d93_a549_4046_8375_15e54524a8da.slice/crio-c365231cb7c45b5784a684fcf54100429aa7c90faa2c6339e629353323d2da37 WatchSource:0}: Error finding container c365231cb7c45b5784a684fcf54100429aa7c90faa2c6339e629353323d2da37: Status 404 returned error can't find the container with id c365231cb7c45b5784a684fcf54100429aa7c90faa2c6339e629353323d2da37
Oct 31 00:26:46.689950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.689482    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:46.689950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.689567    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:46.689950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:46.689609    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:26:47.132166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.132023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" event={"ID":"8b949d93-a549-4046-8375-15e54524a8da","Type":"ContainerStarted","Data":"0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18"}
Oct 31 00:26:47.132539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.132512    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" event={"ID":"8b949d93-a549-4046-8375-15e54524a8da","Type":"ContainerStarted","Data":"c365231cb7c45b5784a684fcf54100429aa7c90faa2c6339e629353323d2da37"}
Oct 31 00:26:47.134987 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.134956    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" event={"ID":"10975acb-c27f-4df0-9c75-61e250866384","Type":"ContainerStarted","Data":"eecbc4f68e6cc78885a6737d43c855f9c4fb9ed304d771f926f02712c1b27985"}
Oct 31 00:26:47.135153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.135011    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" event={"ID":"10975acb-c27f-4df0-9c75-61e250866384","Type":"ContainerStarted","Data":"9b932afcd126b835c7829a1885a90c3db544a6ba6fa1f6860a25160010407574"}
Oct 31 00:26:47.166951 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.166882    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podStartSLOduration=77.166860653 podStartE2EDuration="1m17.166860653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:47.164154763 +0000 UTC m=+1975.110525481" watchObservedRunningTime="2026-10-31 00:26:47.166860653 +0000 UTC m=+1975.113231371"
Oct 31 00:26:47.207196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.207131    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podStartSLOduration=52.207113531 podStartE2EDuration="52.207113531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:47.197933618 +0000 UTC m=+1975.144304337" watchObservedRunningTime="2026-10-31 00:26:47.207113531 +0000 UTC m=+1975.153484269"
Oct 31 00:26:47.261512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.261078    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:47.262978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.262948    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:47.263175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.263024    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:47.335893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.335860    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:47.338886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.338855    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:47.339018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:47.338910    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:48.527488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:48.401684    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:48.858788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:48.858684    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:48.863489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:48.522647    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3aadd02-d322-476b-893b-2459b5f12808" path="/var/lib/kubelet/pods/f3aadd02-d322-476b-893b-2459b5f12808/volumes"
Oct 31 00:26:48.916528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:48.406156    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:48.916528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:48.872032    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:49.261306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:49.261275    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:49.261919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:49.261569    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:49.336062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:49.336027    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:49.336379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:49.336339    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:50.260433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:50.260399    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:50.260901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:50.260445    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:50.336148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:50.335509    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:50.336148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:50.335570    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:51.261353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:51.261320    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:51.261902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:51.261558    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:51.338873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:51.338838    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:51.339143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:51.339112    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:52.262781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:52.262737    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:52.263362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:52.263331    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:52.360064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:52.357740    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:52.360064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:52.357802    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:53.298708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:53.298664    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:53.299319 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:53.299284    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:53.490906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:53.490842    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:53.491223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:53.491192    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:53.767572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:53.746829    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"]
Oct 31 00:26:53.767572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:53.747283    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" podUID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" containerName="maas-api" containerID="cri-o://3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c" gracePeriod=30
Oct 31 00:26:54.025222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.001252    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 31 00:26:54.216488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.200768    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.272488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.224181    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.319294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.313436    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 31 00:26:54.433517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.426312    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.433517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.430330    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 31 00:26:54.433517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.430723    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:54.433517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.430771    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:54.455108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.438251    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.535694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.535650    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:26:54.596114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.596006    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.629539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.629451    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 31 00:26:54.679489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.634554    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.679489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.639204    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 00:26:54.717489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.709020    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 31 00:26:54.811490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.809089    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"]
Oct 31 00:26:54.815484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.812563    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.815484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.814387    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" podUID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.180:8443/health\": dial tcp 10.132.0.180:8443: connect: connection refused"
Oct 31 00:26:54.820486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.817515    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:54.820486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.819387    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"]
Oct 31 00:26:54.820486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.819508    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:54.865717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.863523    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-a8d8bf-dockercfg-lbjtq\""
Oct 31 00:26:54.865717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.863625    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-a8d8bf\""
Oct 31 00:26:54.923487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.906968    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5xf\" (UniqueName: \"kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf\") pod \"payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:54.923823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:54.923794    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:55.031771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.031251    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5xf\" (UniqueName: \"kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf\") pod \"payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:55.031771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.031321    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:55.032501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.032428    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:55.067548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.063383    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5xf\" (UniqueName: \"kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf\") pod \"payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:55.154008 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:55.153947    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac910e3_9861_4a2a_b293_e7ceb5df4e35.slice/crio-3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:26:55.237109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.236866    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:55.338804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.338763    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 31 00:26:55.339337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.338826    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 31 00:26:55.459716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.452446    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:26:55.522417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.521250    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"]
Oct 31 00:26:55.525841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.524395    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" containerName="maas-api"
Oct 31 00:26:55.525841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.524422    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" containerName="maas-api"
Oct 31 00:26:55.525841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.524715    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" containerName="maas-api"
Oct 31 00:26:55.531171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.531153    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.543500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.538579    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08673d2c\""
Oct 31 00:26:55.545606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.544104    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq\" (UniqueName: \"kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq\") pod \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") "
Oct 31 00:26:55.545606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.544216    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs\") pod \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") "
Oct 31 00:26:55.545606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.544350    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca\") pod \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") "
Oct 31 00:26:55.545606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.544497    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls\") pod \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\" (UID: \"0ac910e3-9861-4a2a-b293-e7ceb5df4e35\") "
Oct 31 00:26:55.545606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.544876    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.545606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.544992    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvrn\" (UniqueName: \"kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.550519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.550139    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca" pod "0ac910e3-9861-4a2a-b293-e7ceb5df4e35" (UID: "0ac910e3-9861-4a2a-b293-e7ceb5df4e35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:55.594709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.550899    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"]
Oct 31 00:26:55.594709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.587520    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08673d2c-dockercfg-x9zz8\""
Oct 31 00:26:55.594709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.592516    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls" pod "0ac910e3-9861-4a2a-b293-e7ceb5df4e35" (UID: "0ac910e3-9861-4a2a-b293-e7ceb5df4e35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:26:55.594709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.592664    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq" pod "0ac910e3-9861-4a2a-b293-e7ceb5df4e35" (UID: "0ac910e3-9861-4a2a-b293-e7ceb5df4e35"). InnerVolumeSpecName "kube-api-access-dlrfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:55.594709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.592730    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs" pod "0ac910e3-9861-4a2a-b293-e7ceb5df4e35" (UID: "0ac910e3-9861-4a2a-b293-e7ceb5df4e35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:26:55.605400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.605265    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"]
Oct 31 00:26:55.605400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.605336    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"]
Oct 31 00:26:55.606140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.605762    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.614334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.611667    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:55.614334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.613119    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:55.624844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.623083    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:26:55.646969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.646934    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.647151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647090    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvrn\" (UniqueName: \"kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.647233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647196    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9r2\" (UniqueName: \"kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2\") pod \"payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.647563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647542    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.647676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647661    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:55.647741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647679    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrfq\" (UniqueName: \"kubernetes.io/projected/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-kube-api-access-dlrfq\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:55.647741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647693    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:55.647741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.647735    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac910e3-9861-4a2a-b293-e7ceb5df4e35-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:55.648606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.648579    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.691059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.690905    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvrn\" (UniqueName: \"kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.751727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.751329    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.751727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.751598    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9r2\" (UniqueName: \"kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2\") pod \"payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.756499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.753315    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.780243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.777172    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9r2\" (UniqueName: \"kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2\") pod \"payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:55.811495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.808549    5078 generic.go:356] "Generic (PLEG): container finished" podID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" containerID="3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c" exitCode=0
Oct 31 00:26:55.811495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.809805    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"
Oct 31 00:26:55.813667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.813251    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" event={"ID":"0ac910e3-9861-4a2a-b293-e7ceb5df4e35","Type":"ContainerDied","Data":"3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c"}
Oct 31 00:26:55.813667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.813308    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq" event={"ID":"0ac910e3-9861-4a2a-b293-e7ceb5df4e35","Type":"ContainerDied","Data":"20ef0c350f5e8d84686fb7370d751a96d21098b083a4d3fa8f106d7bda352a71"}
Oct 31 00:26:55.813667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.813337    5078 scope.go:122] "RemoveContainer" containerID="3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c"
Oct 31 00:26:55.867424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.867399    5078 scope.go:122] "RemoveContainer" containerID="3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c"
Oct 31 00:26:55.868571 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:26:55.867969    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c\": container with ID starting with 3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c not found: ID does not exist" containerID="3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c"
Oct 31 00:26:55.868571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.868004    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c"} err="failed to get container status \"3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c\": rpc error: code = NotFound desc = could not find container \"3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c\": container with ID starting with 3b38b28146f47a2100bae8f3fc034de3c2ad9b15b2c2e6c123b0c4b65d59fd9c not found: ID does not exist"
Oct 31 00:26:55.908083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.907978    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"]
Oct 31 00:26:55.914689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.914664    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b786fc-d7557f8ff-g5jhq"]
Oct 31 00:26:55.929840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.929175    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:26:55.930015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.929995    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"]
Oct 31 00:26:55.943491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:55.943394    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:56.215448 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.215413    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac910e3-9861-4a2a-b293-e7ceb5df4e35" path="/var/lib/kubelet/pods/0ac910e3-9861-4a2a-b293-e7ceb5df4e35/volumes"
Oct 31 00:26:56.342864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.342833    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:56.343596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.343570    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:56.344708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.344690    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:26:56.377093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.377065    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"]
Oct 31 00:26:56.488697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.487771    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"]
Oct 31 00:26:56.491780 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:26:56.491612    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b35096f_b88a_4ef8_8280_1ff7b12f2868.slice/crio-f3f0064abc0649499001d66479640726391849d8d4da525bd22631e83be0055d WatchSource:0}: Error finding container f3f0064abc0649499001d66479640726391849d8d4da525bd22631e83be0055d: Status 404 returned error can't find the container with id f3f0064abc0649499001d66479640726391849d8d4da525bd22631e83be0055d
Oct 31 00:26:56.821368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.821273    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs" event={"ID":"7307d862-3d93-4cd1-99b4-2c46b74b3e44","Type":"ContainerStarted","Data":"ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f"}
Oct 31 00:26:56.821678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.821642    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs" event={"ID":"7307d862-3d93-4cd1-99b4-2c46b74b3e44","Type":"ContainerStarted","Data":"a59f15a80a67ce8d319b33dfade331e6fecfec946ce55f2d4a7e4d9ebfe92137"}
Oct 31 00:26:56.821823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.821693    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:26:56.823224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.823198    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj" event={"ID":"f0db0989-f9e7-40d6-bf01-53587d33c3a4","Type":"ContainerStarted","Data":"3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd"}
Oct 31 00:26:56.823338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.823249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj" event={"ID":"f0db0989-f9e7-40d6-bf01-53587d33c3a4","Type":"ContainerStarted","Data":"87dc67dfb400afc691e459d5917772e0aa649c7e6271dfa187a68dc52d2961d1"}
Oct 31 00:26:56.823338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.823300    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:26:56.826213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.826184    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d" event={"ID":"7b35096f-b88a-4ef8-8280-1ff7b12f2868","Type":"ContainerStarted","Data":"ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958"}
Oct 31 00:26:56.826376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.826347    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d" event={"ID":"7b35096f-b88a-4ef8-8280-1ff7b12f2868","Type":"ContainerStarted","Data":"f3f0064abc0649499001d66479640726391849d8d4da525bd22631e83be0055d"}
Oct 31 00:26:56.846165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.846115    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs" podStartSLOduration=3.846099681 podStartE2EDuration="3.846099681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:56.842381944 +0000 UTC m=+1984.788752662" watchObservedRunningTime="2026-10-31 00:26:56.846099681 +0000 UTC m=+1984.792470395"
Oct 31 00:26:56.878218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.878168    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj" podStartSLOduration=3.8781556090000002 podStartE2EDuration="3.878155609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:56.874646748 +0000 UTC m=+1984.821017487" watchObservedRunningTime="2026-10-31 00:26:56.878155609 +0000 UTC m=+1984.824526322"
Oct 31 00:26:56.900553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:56.900499    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d" podStartSLOduration=8.900482754 podStartE2EDuration="8.900482754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:56.897756182 +0000 UTC m=+1984.844126899" watchObservedRunningTime="2026-10-31 00:26:56.900482754 +0000 UTC m=+1984.846853470"
Oct 31 00:26:57.840550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:26:57.840516    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:27:01.521653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:01.521116    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"]
Oct 31 00:27:01.521653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:01.521375    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" containerID="cri-o://c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd" gracePeriod=30
Oct 31 00:27:01.529796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:01.529771    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"]
Oct 31 00:27:01.530090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:01.530043    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" containerID="cri-o://91c80b054825fda4518faf3f81549cc2db58d74b08260b5687b87553c1062e15" gracePeriod=30
Oct 31 00:27:01.538910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:01.538860    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"]
Oct 31 00:27:01.539141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:01.539115    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy" containerID="cri-o://eecbc4f68e6cc78885a6737d43c855f9c4fb9ed304d771f926f02712c1b27985" gracePeriod=30
Oct 31 00:27:06.009241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:06.009204    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:27:06.009669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:06.009261    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:27:06.423321 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:06.423289    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:27:07.174680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.174303    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223230    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223380    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223590    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223732    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223801    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223874    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.223978    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.224019    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.224059    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs\") pod \"3db510f5-171b-43d9-9afc-6abc92243054\" (UID: \"3db510f5-171b-43d9-9afc-6abc92243054\") "
Oct 31 00:27:07.225340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.225083    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.227267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.226425    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.227267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.226745    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:07.227267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.226983    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.227790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.227767    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.228474 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.228430    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.229175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.229138    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "kube-api-access-bplqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:07.233629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.233602    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:07.235711 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.235665    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo" pod "3db510f5-171b-43d9-9afc-6abc92243054" (UID: "3db510f5-171b-43d9-9afc-6abc92243054"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:07.309966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.309872    5078 generic.go:356] "Generic (PLEG): container finished" podID="3db510f5-171b-43d9-9afc-6abc92243054" containerID="c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd" exitCode=0
Oct 31 00:27:07.310136 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.310055    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" event={"ID":"3db510f5-171b-43d9-9afc-6abc92243054","Type":"ContainerDied","Data":"c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd"}
Oct 31 00:27:07.310136 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.310121    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" event={"ID":"3db510f5-171b-43d9-9afc-6abc92243054","Type":"ContainerDied","Data":"c80e1e7d586373c5ff98d2b3b68a1e5170e9fa926a8adf34259b1a0e314df028"}
Oct 31 00:27:07.310261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.310153    5078 scope.go:122] "RemoveContainer" containerID="c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd"
Oct 31 00:27:07.310843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.310357    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"
Oct 31 00:27:07.327666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.326852    5078 generic.go:356] "Generic (PLEG): container finished" podID="10975acb-c27f-4df0-9c75-61e250866384" containerID="eecbc4f68e6cc78885a6737d43c855f9c4fb9ed304d771f926f02712c1b27985" exitCode=0
Oct 31 00:27:07.327666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327156    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" event={"ID":"10975acb-c27f-4df0-9c75-61e250866384","Type":"ContainerDied","Data":"eecbc4f68e6cc78885a6737d43c855f9c4fb9ed304d771f926f02712c1b27985"}
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327677    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327701    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327716    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bplqv\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-kube-api-access-bplqv\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327731    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db510f5-171b-43d9-9afc-6abc92243054-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327745    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327757    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db510f5-171b-43d9-9afc-6abc92243054-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327770    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327782    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db510f5-171b-43d9-9afc-6abc92243054-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.327854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.327796    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db510f5-171b-43d9-9afc-6abc92243054-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.330079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.329955    5078 generic.go:356] "Generic (PLEG): container finished" podID="97548888-626e-4da5-b1c8-45b930ea1e72" containerID="91c80b054825fda4518faf3f81549cc2db58d74b08260b5687b87553c1062e15" exitCode=0
Oct 31 00:27:07.330079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.330024    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" event={"ID":"97548888-626e-4da5-b1c8-45b930ea1e72","Type":"ContainerDied","Data":"91c80b054825fda4518faf3f81549cc2db58d74b08260b5687b87553c1062e15"}
Oct 31 00:27:07.350683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.350579    5078 scope.go:122] "RemoveContainer" containerID="c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd"
Oct 31 00:27:07.365199 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:07.364006    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd\": container with ID starting with c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd not found: ID does not exist" containerID="c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd"
Oct 31 00:27:07.365199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.364051    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd"} err="failed to get container status \"c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd\": rpc error: code = NotFound desc = could not find container \"c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd\": container with ID starting with c1ecd2fe8d8b7f3bf8195ef11c5bb7b4088b2fbd56a3dd1a46390cd329e498bd not found: ID does not exist"
Oct 31 00:27:07.365401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.365245    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:27:07.368135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.368113    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:27:07.382134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.381904    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"]
Oct 31 00:27:07.385638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.385606    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl"]
Oct 31 00:27:07.428771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428741    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.428963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428788    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.428963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428845    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.428963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428864    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.428963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428913    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.428963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428930    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.428973    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429092    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429112    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429135    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429159    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429187    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429308    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429350    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429378    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429453    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo\") pod \"10975acb-c27f-4df0-9c75-61e250866384\" (UID: \"10975acb-c27f-4df0-9c75-61e250866384\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429509    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo\") pod \"97548888-626e-4da5-b1c8-45b930ea1e72\" (UID: \"97548888-626e-4da5-b1c8-45b930ea1e72\") "
Oct 31 00:27:07.430105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.429652    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.431032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.430952    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.431659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.431147    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.431659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.431345    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.431659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.431477    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.431869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.431784    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.432236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.432208    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:07.432640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.432613    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:07.433311 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.433289    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.434092 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.434063    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.437211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.437183    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:07.437316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.437264    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.437500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.437452    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.437593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.437493    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:07.437750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.437726    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn" pod "10975acb-c27f-4df0-9c75-61e250866384" (UID: "10975acb-c27f-4df0-9c75-61e250866384"). InnerVolumeSpecName "kube-api-access-tsxfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:07.438665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.438641    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "kube-api-access-2cbhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:07.439079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.439051    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:07.439731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.439661    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:07.441716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.441674    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token" pod "97548888-626e-4da5-b1c8-45b930ea1e72" (UID: "97548888-626e-4da5-b1c8-45b930ea1e72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:07.539077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539040    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10975acb-c27f-4df0-9c75-61e250866384-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539077    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97548888-626e-4da5-b1c8-45b930ea1e72-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539087    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539099    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539108    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539116    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539125    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539133    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10975acb-c27f-4df0-9c75-61e250866384-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539142    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539151    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539159    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97548888-626e-4da5-b1c8-45b930ea1e72-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539167    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10975acb-c27f-4df0-9c75-61e250866384-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539174    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539184    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxfn\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-kube-api-access-tsxfn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539193    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97548888-626e-4da5-b1c8-45b930ea1e72-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539200    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10975acb-c27f-4df0-9c75-61e250866384-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.539288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.539211    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbhh\" (UniqueName: \"kubernetes.io/projected/97548888-626e-4da5-b1c8-45b930ea1e72-kube-api-access-2cbhh\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:07.843672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.843365    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:27:07.843846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.843706    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:27:07.911624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.911588    5078 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:27:07.911925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:07.911895    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-9p6fl" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:08.203338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.203244    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db510f5-171b-43d9-9afc-6abc92243054" path="/var/lib/kubelet/pods/3db510f5-171b-43d9-9afc-6abc92243054/volumes"
Oct 31 00:27:08.356483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.352078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"
Oct 31 00:27:08.356483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.352165    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb" event={"ID":"97548888-626e-4da5-b1c8-45b930ea1e72","Type":"ContainerDied","Data":"dd31724fa4e2f0bb9b5d732c07e4097fc22ac7c126a5050009fd835bb0e9e2c9"}
Oct 31 00:27:08.356483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.353006    5078 scope.go:122] "RemoveContainer" containerID="91c80b054825fda4518faf3f81549cc2db58d74b08260b5687b87553c1062e15"
Oct 31 00:27:08.368197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.368174    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"
Oct 31 00:27:08.368197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.368184    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq" event={"ID":"10975acb-c27f-4df0-9c75-61e250866384","Type":"ContainerDied","Data":"9b932afcd126b835c7829a1885a90c3db544a6ba6fa1f6860a25160010407574"}
Oct 31 00:27:08.480548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.454959    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"]
Oct 31 00:27:08.481648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.476413    5078 scope.go:122] "RemoveContainer" containerID="eecbc4f68e6cc78885a6737d43c855f9c4fb9ed304d771f926f02712c1b27985"
Oct 31 00:27:08.498598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.498567    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-w2tlb"]
Oct 31 00:27:08.506938 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.506909    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"]
Oct 31 00:27:08.515382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.515356    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-kbjjq"]
Oct 31 00:27:08.858428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:08.858346    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:27:09.026539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.026505    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"]
Oct 31 00:27:09.042067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.042038    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260cf387-7t7th"]
Oct 31 00:27:09.056336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.056303    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"]
Oct 31 00:27:09.057879 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.057852    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy"
Oct 31 00:27:09.057879 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.057879    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy"
Oct 31 00:27:09.058100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.057897    5078 cpu_manager.go:414] "containerMap: removing container" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy"
Oct 31 00:27:09.058100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.057905    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy"
Oct 31 00:27:09.058100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.057963    5078 cpu_manager.go:414] "containerMap: removing container" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy"
Oct 31 00:27:09.058100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.057971    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy"
Oct 31 00:27:09.058481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.058212    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" containerName="istio-proxy"
Oct 31 00:27:09.058481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.058239    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db510f5-171b-43d9-9afc-6abc92243054" containerName="istio-proxy"
Oct 31 00:27:09.058481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.058249    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="10975acb-c27f-4df0-9c75-61e250866384" containerName="istio-proxy"
Oct 31 00:27:09.082789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.082758    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"]
Oct 31 00:27:09.082927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.082879    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.088658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.088633    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:27:09.089004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.088984    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:27:09.211602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.211567    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.211977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.211809    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztxf\" (UniqueName: \"kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.212017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.211994    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.313433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.313403    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.314574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.313542    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztxf\" (UniqueName: \"kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.314574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.313634    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.314574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.314301    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.316623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.316595    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.327720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.327693    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztxf\" (UniqueName: \"kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf\") pod \"maas-api-revoke-keys-e2e-api-a-719e12-29v5p\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.403820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.403794    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:09.810239 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:27:09.810204    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94bba3f_f9b9_4d3f_8bc3_7fccc8658e27.slice/crio-9a8dd319e90aad47ec47a5dff753a40dc454d68a5b8fe80359b5b1c16098110b WatchSource:0}: Error finding container 9a8dd319e90aad47ec47a5dff753a40dc454d68a5b8fe80359b5b1c16098110b: Status 404 returned error can't find the container with id 9a8dd319e90aad47ec47a5dff753a40dc454d68a5b8fe80359b5b1c16098110b
Oct 31 00:27:09.811142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:09.811114    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"]
Oct 31 00:27:10.202096 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:10.202058    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10975acb-c27f-4df0-9c75-61e250866384" path="/var/lib/kubelet/pods/10975acb-c27f-4df0-9c75-61e250866384/volumes"
Oct 31 00:27:10.202933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:10.202841    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8578bae2-cc5d-40a6-87b2-19c129319314" path="/var/lib/kubelet/pods/8578bae2-cc5d-40a6-87b2-19c129319314/volumes"
Oct 31 00:27:10.203504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:10.203475    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97548888-626e-4da5-b1c8-45b930ea1e72" path="/var/lib/kubelet/pods/97548888-626e-4da5-b1c8-45b930ea1e72/volumes"
Oct 31 00:27:10.400683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:10.400640    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p" event={"ID":"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27","Type":"ContainerStarted","Data":"9f900d76591805fb4458f5052a9906e3b1b60d1983c442380e3d516fed527053"}
Oct 31 00:27:10.401189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:10.400753    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p" event={"ID":"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27","Type":"ContainerStarted","Data":"9a8dd319e90aad47ec47a5dff753a40dc454d68a5b8fe80359b5b1c16098110b"}
Oct 31 00:27:10.428835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:10.428782    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p" podStartSLOduration=1.428766207 podStartE2EDuration="1.428766207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:10.422328492 +0000 UTC m=+1998.368699203" watchObservedRunningTime="2026-10-31 00:27:10.428766207 +0000 UTC m=+1998.375136923"
Oct 31 00:27:11.412061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:11.411972    5078 generic.go:356] "Generic (PLEG): container finished" podID="a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" containerID="9f900d76591805fb4458f5052a9906e3b1b60d1983c442380e3d516fed527053" exitCode=0
Oct 31 00:27:11.412593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:11.412066    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p" event={"ID":"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27","Type":"ContainerDied","Data":"9f900d76591805fb4458f5052a9906e3b1b60d1983c442380e3d516fed527053"}
Oct 31 00:27:12.080520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.080475    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"]
Oct 31 00:27:12.122744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.122702    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"]
Oct 31 00:27:12.122912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.122859    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.131581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.131554    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cc0fe1\""
Oct 31 00:27:12.131730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.131665    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cc0fe1-dockercfg-fc9vw\""
Oct 31 00:27:12.218511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.218183    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cmv\" (UniqueName: \"kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv\") pod \"payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.218511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.218389    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspfm\" (UniqueName: \"kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm\") pod \"payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.218777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.218532    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.218777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.218623    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.412488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.411853    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.412488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.411970    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.412488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.412227    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cmv\" (UniqueName: \"kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv\") pod \"payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.412488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.412374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vspfm\" (UniqueName: \"kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm\") pod \"payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.442865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.440993    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.446484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.443224    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.620862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.535668    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspfm\" (UniqueName: \"kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm\") pod \"payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:12.620862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.539122    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.620862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.546690    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cmv\" (UniqueName: \"kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv\") pod \"payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.658430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.658403    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"]
Oct 31 00:27:12.658760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.658747    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"]
Oct 31 00:27:12.783381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.783283    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:12.801685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:12.800728    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:13.156303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.151150    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"]
Oct 31 00:27:13.193569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.193537    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"]
Oct 31 00:27:13.194282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.193683    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.207213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.207188    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:13.216927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.216902    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-a8d8bf-serving-cert\""
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.286065    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf\" (UniqueName: \"kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf\") pod \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") "
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.286264    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca\") pod \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") "
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.286480    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca\") pod \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\" (UID: \"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27\") "
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.286929    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.287085    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dnk\" (UniqueName: \"kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.287214    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.287278    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.290581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.288549    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca" pod "a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" (UID: "a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:13.405409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.391901    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dnk\" (UniqueName: \"kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.405409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.392051    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.405409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.392118    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.405409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.392423    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.405409 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:13.393191    5078 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-serving-cert: secret "maas-api-e2e-shared-b-w5-a8d8bf-serving-cert" not found
Oct 31 00:27:13.405409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.402291    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.413620 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:13.405618    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls podName:b32cf632-468b-4ffc-a143-4ff9d703a3e6 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:13.893264931 +0000 UTC m=+2001.839635631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls") pod "maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" (UID: "b32cf632-468b-4ffc-a143-4ff9d703a3e6") : secret "maas-api-e2e-shared-b-w5-a8d8bf-serving-cert" not found
Oct 31 00:27:13.423279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.422682    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:13.423279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.422685    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca" pod "a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" (UID: "a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:13.423279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.423234    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf" pod "a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" (UID: "a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27"). InnerVolumeSpecName "kube-api-access-bztxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:13.431522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.427012    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.437078 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.436998    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dnk\" (UniqueName: \"kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.503267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.503228    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"]
Oct 31 00:27:13.534349 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:27:13.518815    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fa4ee3_9bfd_43ac_97c6_9448a4635262.slice/crio-f9d30cac3a60ccfeff6394bd44a8242def1262095dd3fef8f416df2dea96f632 WatchSource:0}: Error finding container f9d30cac3a60ccfeff6394bd44a8242def1262095dd3fef8f416df2dea96f632: Status 404 returned error can't find the container with id f9d30cac3a60ccfeff6394bd44a8242def1262095dd3fef8f416df2dea96f632
Oct 31 00:27:13.534349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.527406    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztxf\" (UniqueName: \"kubernetes.io/projected/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-kube-api-access-bztxf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:13.534349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.527430    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:13.627593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.626346    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"]
Oct 31 00:27:13.656224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.656074    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"
Oct 31 00:27:13.656224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.656092    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p" event={"ID":"a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27","Type":"ContainerDied","Data":"9a8dd319e90aad47ec47a5dff753a40dc454d68a5b8fe80359b5b1c16098110b"}
Oct 31 00:27:13.656224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.656151    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a8dd319e90aad47ec47a5dff753a40dc454d68a5b8fe80359b5b1c16098110b"
Oct 31 00:27:13.657790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.657727    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4" event={"ID":"c0429edc-ac06-4abe-8811-31047ba93cf4","Type":"ContainerStarted","Data":"dc7989c4eff0862a338bc466c2985b4635a1fa393006d745a329022d33cadda6"}
Oct 31 00:27:13.659105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.659062    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb" event={"ID":"55fa4ee3-9bfd-43ac-97c6-9448a4635262","Type":"ContainerStarted","Data":"f9d30cac3a60ccfeff6394bd44a8242def1262095dd3fef8f416df2dea96f632"}
Oct 31 00:27:13.945074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.942220    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:13.948614 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:13.948556    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:14.166565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.166532    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:14.592103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.592016    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"]
Oct 31 00:27:14.628552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.628527    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:27:14.673579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.673536    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb" event={"ID":"55fa4ee3-9bfd-43ac-97c6-9448a4635262","Type":"ContainerStarted","Data":"3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451"}
Oct 31 00:27:14.673776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.673673    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:14.675978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.675939    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" event={"ID":"b32cf632-468b-4ffc-a143-4ff9d703a3e6","Type":"ContainerStarted","Data":"7fad2f70616e66183b768a42277d978d43a285714ac7f66dcae359b400fd2287"}
Oct 31 00:27:14.678223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.678192    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4" event={"ID":"c0429edc-ac06-4abe-8811-31047ba93cf4","Type":"ContainerStarted","Data":"20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424"}
Oct 31 00:27:14.679074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.679058    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:14.699051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.699000    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb" podStartSLOduration=2.698983766 podStartE2EDuration="2.698983766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:14.696410461 +0000 UTC m=+2002.642781177" watchObservedRunningTime="2026-10-31 00:27:14.698983766 +0000 UTC m=+2002.645354484"
Oct 31 00:27:14.724267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:14.724213    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4" podStartSLOduration=2.724195897 podStartE2EDuration="2.724195897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:14.719743549 +0000 UTC m=+2002.666114267" watchObservedRunningTime="2026-10-31 00:27:14.724195897 +0000 UTC m=+2002.670566615"
Oct 31 00:27:15.011651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:15.011605    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:27:15.011852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:15.011696    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:18.810371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:18.810326    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" event={"ID":"b32cf632-468b-4ffc-a143-4ff9d703a3e6","Type":"ContainerStarted","Data":"7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612"}
Oct 31 00:27:18.810847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:18.810436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:18.839086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:18.839025    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" podStartSLOduration=2.705918484 podStartE2EDuration="5.839010469s" totalImagesPullingTime="3.133091985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:18.83228233 +0000 UTC m=+2006.778653043" watchObservedRunningTime="2026-10-31 00:27:18.839010469 +0000 UTC m=+2006.785381222"
Oct 31 00:27:19.437086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:19.437050    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"]
Oct 31 00:27:20.964930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:20.964893    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"]
Oct 31 00:27:20.967830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:20.967777    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" containerName="revoke-keys"
Oct 31 00:27:20.967830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:20.967808    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" containerName="revoke-keys"
Oct 31 00:27:20.968194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:20.968173    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" containerName="revoke-keys"
Oct 31 00:27:20.994488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:20.973076    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:20.994488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:20.981616    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d898f3-openshift-default-dockercfg-6mnlq\""
Oct 31 00:27:21.010316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.009690    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"]
Oct 31 00:27:21.048838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.048800    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.048843    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.048870    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzpc\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.049001    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.049203    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.049244    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.049295    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.049315    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.049342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.049339    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.150589    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151180    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151559 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151540    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151600    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151638    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151679    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151843    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151893    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.151926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151917    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzpc\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.152025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.151960    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.152062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.152036    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.152405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.152377    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.152405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.152402    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.152663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.152535    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.154674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.154651    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.154885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.154862    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.170003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.169981    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.170400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.170378    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzpc\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc\") pod \"e2e-api-b-d898f3-openshift-default-6cc9655-xlf76\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") " pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.312534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.306653    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:21.680103 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:27:21.680075    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5665dbb_ca2f_43f3_b7cd_4f1f1dc257da.slice/crio-079a0810048bcf0fb8174028536b4699460b6e5d166b18928e4c03735d565c74 WatchSource:0}: Error finding container 079a0810048bcf0fb8174028536b4699460b6e5d166b18928e4c03735d565c74: Status 404 returned error can't find the container with id 079a0810048bcf0fb8174028536b4699460b6e5d166b18928e4c03735d565c74
Oct 31 00:27:21.682898 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.682873    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"]
Oct 31 00:27:21.684512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.684445    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:21.684661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.684576    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:21.684661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.684626    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:21.988418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.988319    5078 generic.go:356] "Generic (PLEG): container finished" podID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerID="4f1c2ec1c5698228c54c5c7e7827d66bed3da07fc9f105c2890f011b3b3a1b4e" exitCode=0
Oct 31 00:27:21.988418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.988404    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" event={"ID":"9a603b11-c6d9-4c14-932f-c1e3d1454bea","Type":"ContainerDied","Data":"4f1c2ec1c5698228c54c5c7e7827d66bed3da07fc9f105c2890f011b3b3a1b4e"}
Oct 31 00:27:21.988986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.988963    5078 scope.go:122] "RemoveContainer" containerID="4f1c2ec1c5698228c54c5c7e7827d66bed3da07fc9f105c2890f011b3b3a1b4e"
Oct 31 00:27:21.992179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.992139    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:21.992316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.992242    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:21.992316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.992294    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:21.993872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.993843    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" event={"ID":"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da","Type":"ContainerStarted","Data":"1a630d9284e37575cd1158c5a5edef094977a44e1abe5ed667cc27f72e85717a"}
Oct 31 00:27:21.993983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:21.993899    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" event={"ID":"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da","Type":"ContainerStarted","Data":"079a0810048bcf0fb8174028536b4699460b6e5d166b18928e4c03735d565c74"}
Oct 31 00:27:22.082415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:22.082357    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podStartSLOduration=2.082339777 podStartE2EDuration="2.082339777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:22.074118549 +0000 UTC m=+2010.020489270" watchObservedRunningTime="2026-10-31 00:27:22.082339777 +0000 UTC m=+2010.028710497"
Oct 31 00:27:22.311517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:22.311402    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:27:22.311774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:22.311744    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:27:22.350224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:22.350191    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:23.008423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:23.008378    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" event={"ID":"9a603b11-c6d9-4c14-932f-c1e3d1454bea","Type":"ContainerStarted","Data":"0d1c9edcee13b484d7acbe402615452fb74576dbec66b0c1ca4086158eb34e44"}
Oct 31 00:27:23.314404 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:23.314312    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:23.604542 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:23.603717    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:27:23.608483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:23.605112    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 00:27:23.608483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:23.605167    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 00:27:24.021728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.021689    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:24.022602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.022581    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:27:24.438088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.438037    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn" podUID="f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" containerName="payload-pre-processing" containerID="cri-o://7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa" gracePeriod=25
Oct 31 00:27:24.608424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.608378    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 00:27:24.608715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.608511    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 00:27:24.924485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.921098    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:27:24.936279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:24.936254    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:27:25.016596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.012325    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx\" (UniqueName: \"kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx\") pod \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") "
Oct 31 00:27:25.017279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.017243    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume\") pod \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\" (UID: \"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab\") "
Oct 31 00:27:25.019550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.019223    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume" pod "f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" (UID: "f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:25.020494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.020473    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:25.023844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.023808    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx" pod "f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" (UID: "f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab"). InnerVolumeSpecName "kube-api-access-dknxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:25.040832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.040803    5078 generic.go:356] "Generic (PLEG): container finished" podID="f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" containerID="7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa" exitCode=0
Oct 31 00:27:25.041956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.041935    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"
Oct 31 00:27:25.042372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.042107    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn" event={"ID":"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab","Type":"ContainerDied","Data":"7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa"}
Oct 31 00:27:25.042372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.042179    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn" event={"ID":"f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab","Type":"ContainerDied","Data":"d09fc0bb3e052a2e8c9995dae31ccb6dbac9a364e18eaf8a2dbd2757380087b7"}
Oct 31 00:27:25.042584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.042370    5078 scope.go:122] "RemoveContainer" containerID="7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa"
Oct 31 00:27:25.137339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.130685    5078 scope.go:122] "RemoveContainer" containerID="7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa"
Oct 31 00:27:25.137339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.132895    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dknxx\" (UniqueName: \"kubernetes.io/projected/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab-kube-api-access-dknxx\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:25.138862 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:25.138829    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa\": container with ID starting with 7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa not found: ID does not exist" containerID="7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa"
Oct 31 00:27:25.139044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.139018    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa"} err="failed to get container status \"7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa\": rpc error: code = NotFound desc = could not find container \"7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa\": container with ID starting with 7201c845813250d5e18a09045d59148da39bebc0676b3338f2eaa33c2b1e24aa not found: ID does not exist"
Oct 31 00:27:25.144750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.144727    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"]
Oct 31 00:27:25.152384 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.152356    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-719e12-665f9cb46c-5x8fn"]
Oct 31 00:27:25.617487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.611877    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 00:27:25.617487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.611933    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 00:27:25.718489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.716830    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:27:25.731509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:25.719333    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:27:26.216596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:26.216564    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" path="/var/lib/kubelet/pods/f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab/volumes"
Oct 31 00:27:26.604228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:26.604136    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 00:27:26.604577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:26.604542    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 00:27:27.618665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:27.615264    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 00:27:27.618665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:27.615323    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 00:27:28.629491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:28.609343    5078 patch_prober.go:29] interesting pod/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 00:27:28.630198 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:28.630162    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 00:27:28.965622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:28.965484    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:27:29.725455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:29.725416    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:27:29.726942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:29.726918    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:27:30.342843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:30.342810    5078 scope.go:122] "RemoveContainer" containerID="2b01d87a1755dcf35835c595c30550a6872ae58269423156f68be84f30801498"
Oct 31 00:27:30.361996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:30.361971    5078 scope.go:122] "RemoveContainer" containerID="df1304ea316ce05791751fed5c6ec9eb298e602c88d5f94b5da50d22e771736e"
Oct 31 00:27:30.376142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:30.376123    5078 scope.go:122] "RemoveContainer" containerID="e4bceaf7b3b5945523f6f9dd8e30e6993af19f8e8e15a57bdc9b828cb5b3d4ab"
Oct 31 00:27:34.118753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.118667    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"]
Oct 31 00:27:34.120218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.120192    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" containerName="payload-pre-processing"
Oct 31 00:27:34.120313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.120221    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" containerName="payload-pre-processing"
Oct 31 00:27:34.120498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.120447    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1977d25-4dd8-4ad7-adc5-1f7b0b5815ab" containerName="payload-pre-processing"
Oct 31 00:27:34.127820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.127794    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.135297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.135264    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"]
Oct 31 00:27:34.147266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.147237    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:27:34.168531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.149526    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:27:34.280040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.279996    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.280247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.280157    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.280320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.280239    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjsf\" (UniqueName: \"kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.383477 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.383368    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.383890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.383866    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.384800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.384774    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjsf\" (UniqueName: \"kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.386631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.386258    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.400491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.399207    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.444057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.441552    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjsf\" (UniqueName: \"kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf\") pod \"maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.477515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.470897    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:34.960813 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:27:34.960740    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21ffa9f_c28d_45df_a0a5_4dc82f719693.slice/crio-7ca06c704f964203f625d57aff84bcaea03a641180f84decd7b3f61ef38c86b4 WatchSource:0}: Error finding container 7ca06c704f964203f625d57aff84bcaea03a641180f84decd7b3f61ef38c86b4: Status 404 returned error can't find the container with id 7ca06c704f964203f625d57aff84bcaea03a641180f84decd7b3f61ef38c86b4
Oct 31 00:27:34.962607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:34.962585    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"]
Oct 31 00:27:35.010390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:35.010363    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"]
Oct 31 00:27:35.010625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:35.010602    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy" containerID="cri-o://f6312f987189427af8c5d0b6ff23e16589b9415380b519078678948536318780" gracePeriod=30
Oct 31 00:27:35.298949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:35.298909    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf" event={"ID":"f21ffa9f-c28d-45df-a0a5-4dc82f719693","Type":"ContainerStarted","Data":"7ca06c704f964203f625d57aff84bcaea03a641180f84decd7b3f61ef38c86b4"}
Oct 31 00:27:36.311176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:36.311140    5078 generic.go:356] "Generic (PLEG): container finished" podID="f21ffa9f-c28d-45df-a0a5-4dc82f719693" containerID="afad5a1c5eed461d98ea6f81c8580063234aa76e8510c795095781654a988615" exitCode=0
Oct 31 00:27:36.311682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:36.311234    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf" event={"ID":"f21ffa9f-c28d-45df-a0a5-4dc82f719693","Type":"ContainerDied","Data":"afad5a1c5eed461d98ea6f81c8580063234aa76e8510c795095781654a988615"}
Oct 31 00:27:36.408650 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:36.408614    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:27:37.842032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:37.841776    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:38.000756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.000651    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca\") pod \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") "
Oct 31 00:27:38.000756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.000748    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca\") pod \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") "
Oct 31 00:27:38.000983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.000839    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf\" (UniqueName: \"kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf\") pod \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\" (UID: \"f21ffa9f-c28d-45df-a0a5-4dc82f719693\") "
Oct 31 00:27:38.001120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.001094    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca" pod "f21ffa9f-c28d-45df-a0a5-4dc82f719693" (UID: "f21ffa9f-c28d-45df-a0a5-4dc82f719693"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:38.001978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.001945    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:38.004146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.004116    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf" pod "f21ffa9f-c28d-45df-a0a5-4dc82f719693" (UID: "f21ffa9f-c28d-45df-a0a5-4dc82f719693"). InnerVolumeSpecName "kube-api-access-8wjsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:38.006608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.006581    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca" pod "f21ffa9f-c28d-45df-a0a5-4dc82f719693" (UID: "f21ffa9f-c28d-45df-a0a5-4dc82f719693"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:38.103970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.103934    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21ffa9f-c28d-45df-a0a5-4dc82f719693-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:38.103970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.103963    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjsf\" (UniqueName: \"kubernetes.io/projected/f21ffa9f-c28d-45df-a0a5-4dc82f719693-kube-api-access-8wjsf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:38.358500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.358376    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf" event={"ID":"f21ffa9f-c28d-45df-a0a5-4dc82f719693","Type":"ContainerDied","Data":"7ca06c704f964203f625d57aff84bcaea03a641180f84decd7b3f61ef38c86b4"}
Oct 31 00:27:38.358500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.358442    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca06c704f964203f625d57aff84bcaea03a641180f84decd7b3f61ef38c86b4"
Oct 31 00:27:38.358712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:38.358523    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"
Oct 31 00:27:40.085486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.084964    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"]
Oct 31 00:27:40.085486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.085229    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy" containerID="cri-o://2eb948f8ec0d61796cf275d27a9c21e292c4ef3dd00beb8674c6aabcbfa4cc7d" gracePeriod=30
Oct 31 00:27:40.094397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.094367    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"]
Oct 31 00:27:40.094709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.094653    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy" containerID="cri-o://7bbb4b7931d9b1edb29049aa378dd4e6dc96d69f7432afc53a7cfb7a810d25e1" gracePeriod=30
Oct 31 00:27:40.106919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.103435    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"]
Oct 31 00:27:40.106919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.103700    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy" containerID="cri-o://60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8" gracePeriod=30
Oct 31 00:27:40.112982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.111773    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"]
Oct 31 00:27:40.112982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.111980    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy" containerID="cri-o://4b3945c907d84a1579b7b3f8fe3e8fb2a8e7914e79f98c3a5e385e9d768cf8f7" gracePeriod=30
Oct 31 00:27:40.413496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.411906    5078 generic.go:356] "Generic (PLEG): container finished" podID="5de1af60-a588-4e7e-b8cc-b201871af022" containerID="f6312f987189427af8c5d0b6ff23e16589b9415380b519078678948536318780" exitCode=0
Oct 31 00:27:40.413496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.411984    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" event={"ID":"5de1af60-a588-4e7e-b8cc-b201871af022","Type":"ContainerDied","Data":"f6312f987189427af8c5d0b6ff23e16589b9415380b519078678948536318780"}
Oct 31 00:27:40.467562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.467433    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:27:40.577330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577292    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.577545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577367    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.577545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577425    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.577687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577666    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.577765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577746    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.577821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577807    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.577914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.577898    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.578042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.578007    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.578218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.578198    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert\") pod \"5de1af60-a588-4e7e-b8cc-b201871af022\" (UID: \"5de1af60-a588-4e7e-b8cc-b201871af022\") "
Oct 31 00:27:40.578601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.578575    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:40.578931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.578906    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:40.579134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.579108    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:40.579625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.579363    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.579625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.579387    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de1af60-a588-4e7e-b8cc-b201871af022-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.579625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.579401    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.580722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.579930    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:40.581611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.580576    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:40.591820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.591792    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:40.592113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.592086    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:40.594190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.594158    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:40.594569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.594535    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk" pod "5de1af60-a588-4e7e-b8cc-b201871af022" (UID: "5de1af60-a588-4e7e-b8cc-b201871af022"). InnerVolumeSpecName "kube-api-access-z75lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:40.681662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.681575    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de1af60-a588-4e7e-b8cc-b201871af022-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.681662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.681608    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.681662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.681622    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.681662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.681634    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.681662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.681647    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z75lk\" (UniqueName: \"kubernetes.io/projected/5de1af60-a588-4e7e-b8cc-b201871af022-kube-api-access-z75lk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:40.681662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:40.681661    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de1af60-a588-4e7e-b8cc-b201871af022-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:41.424456 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:41.424415    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9" event={"ID":"5de1af60-a588-4e7e-b8cc-b201871af022","Type":"ContainerDied","Data":"e1ddf97877753977b92607fd5775ae5a4fa96f38e534caa3d9b27309872a50c6"}
Oct 31 00:27:41.424961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:41.424432    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"
Oct 31 00:27:41.424961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:41.424529    5078 scope.go:122] "RemoveContainer" containerID="f6312f987189427af8c5d0b6ff23e16589b9415380b519078678948536318780"
Oct 31 00:27:41.454646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:41.454619    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"]
Oct 31 00:27:41.459777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:41.459750    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-g42j9"]
Oct 31 00:27:42.233511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:42.233473    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" path="/var/lib/kubelet/pods/5de1af60-a588-4e7e-b8cc-b201871af022/volumes"
Oct 31 00:27:43.578543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.578498    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"]
Oct 31 00:27:43.579989 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.579963    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f21ffa9f-c28d-45df-a0a5-4dc82f719693" containerName="revoke-keys"
Oct 31 00:27:43.579989 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.579989    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21ffa9f-c28d-45df-a0a5-4dc82f719693" containerName="revoke-keys"
Oct 31 00:27:43.580150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.580090    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy"
Oct 31 00:27:43.580150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.580099    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy"
Oct 31 00:27:43.580715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.580337    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21ffa9f-c28d-45df-a0a5-4dc82f719693" containerName="revoke-keys"
Oct 31 00:27:43.580715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.580371    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de1af60-a588-4e7e-b8cc-b201871af022" containerName="istio-proxy"
Oct 31 00:27:43.585016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.584993    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.611201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.611169    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2d9875-openshift-default-dockercfg-dzxjr\""
Oct 31 00:27:43.615877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.615851    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"]
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665291    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665352    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5wt\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665396    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665433    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665504    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665584    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665640    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665892    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.667528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.665991    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.815677    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.815785    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.815941    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.815994    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5wt\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.816035    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.816072    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.816128    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.816206    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.816626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.816262    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.844543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.826448    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.844543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.827452    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.844543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.835827    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.844543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.836533    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.855557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.850120    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.915493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.915077    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.948316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.948272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.951501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.949754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:43.966505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:43.966168    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5wt\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt\") pod \"e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") " pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:44.208644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:44.208598    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:44.300018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:44.298394    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"]
Oct 31 00:27:44.508231 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:27:44.508197    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603f2b77_a5c8_403a_97e9_c143099f8fb3.slice/crio-9adadc9592a1dc471acc238a2920c582522150a68c3ec583f59674bc4e6bc4c1 WatchSource:0}: Error finding container 9adadc9592a1dc471acc238a2920c582522150a68c3ec583f59674bc4e6bc4c1: Status 404 returned error can't find the container with id 9adadc9592a1dc471acc238a2920c582522150a68c3ec583f59674bc4e6bc4c1
Oct 31 00:27:44.510558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:44.510535    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"]
Oct 31 00:27:44.511891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:44.511853    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:44.511976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:44.511940    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:44.512057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:44.511990    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:27:45.494081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.494037    5078 generic.go:356] "Generic (PLEG): container finished" podID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerID="7bbb4b7931d9b1edb29049aa378dd4e6dc96d69f7432afc53a7cfb7a810d25e1" exitCode=0
Oct 31 00:27:45.494524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.494188    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" event={"ID":"bba615fc-186e-48cb-9bf1-18ee9c33dc9a","Type":"ContainerDied","Data":"7bbb4b7931d9b1edb29049aa378dd4e6dc96d69f7432afc53a7cfb7a810d25e1"}
Oct 31 00:27:45.535847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.535808    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" event={"ID":"603f2b77-a5c8-403a-97e9-c143099f8fb3","Type":"ContainerStarted","Data":"444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"}
Oct 31 00:27:45.536031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.535871    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" event={"ID":"603f2b77-a5c8-403a-97e9-c143099f8fb3","Type":"ContainerStarted","Data":"9adadc9592a1dc471acc238a2920c582522150a68c3ec583f59674bc4e6bc4c1"}
Oct 31 00:27:45.538444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.538413    5078 generic.go:356] "Generic (PLEG): container finished" podID="4da9275e-742e-4820-8580-c48195fe6776" containerID="4b3945c907d84a1579b7b3f8fe3e8fb2a8e7914e79f98c3a5e385e9d768cf8f7" exitCode=0
Oct 31 00:27:45.538715 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.538451    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" event={"ID":"4da9275e-742e-4820-8580-c48195fe6776","Type":"ContainerDied","Data":"4b3945c907d84a1579b7b3f8fe3e8fb2a8e7914e79f98c3a5e385e9d768cf8f7"}
Oct 31 00:27:45.540554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.540529    5078 generic.go:356] "Generic (PLEG): container finished" podID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerID="2eb948f8ec0d61796cf275d27a9c21e292c4ef3dd00beb8674c6aabcbfa4cc7d" exitCode=0
Oct 31 00:27:45.540707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.540690    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" event={"ID":"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377","Type":"ContainerDied","Data":"2eb948f8ec0d61796cf275d27a9c21e292c4ef3dd00beb8674c6aabcbfa4cc7d"}
Oct 31 00:27:45.542695 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.542674    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:27:45.543115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.542792    5078 generic.go:356] "Generic (PLEG): container finished" podID="690002f6-4096-42d6-bcf2-250b98c55207" containerID="60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8" exitCode=0
Oct 31 00:27:45.543115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.542824    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" event={"ID":"690002f6-4096-42d6-bcf2-250b98c55207","Type":"ContainerDied","Data":"60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8"}
Oct 31 00:27:45.543115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.542972    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft" event={"ID":"690002f6-4096-42d6-bcf2-250b98c55207","Type":"ContainerDied","Data":"189e77b44d096c9d5c8f9e4d0980735da2cbc11c2ca8ed0968455dfcb8a66a80"}
Oct 31 00:27:45.543115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.543006    5078 scope.go:122] "RemoveContainer" containerID="60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8"
Oct 31 00:27:45.557043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.557023    5078 scope.go:122] "RemoveContainer" containerID="60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8"
Oct 31 00:27:45.557355 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:45.557336    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8\": container with ID starting with 60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8 not found: ID does not exist" containerID="60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8"
Oct 31 00:27:45.557594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.557360    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8"} err="failed to get container status \"60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8\": rpc error: code = NotFound desc = could not find container \"60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8\": container with ID starting with 60b0ee4d56aaaa953858801e93d07da48a3e709694e110872f3328cda3ba05c8 not found: ID does not exist"
Oct 31 00:27:45.576490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.575759    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podStartSLOduration=2.57574027 podStartE2EDuration="2.57574027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:45.566207399 +0000 UTC m=+2033.512578119" watchObservedRunningTime="2026-10-31 00:27:45.57574027 +0000 UTC m=+2033.522110989"
Oct 31 00:27:45.587228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.587205    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:27:45.592553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.592532    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:27:45.598426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.598361    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:27:45.696437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696158    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.696437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696217    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.696437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.696437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696285    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.696851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696724    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.696851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696757    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:45.696976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.696954    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697085    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.697164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697143    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697176    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697237    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.697366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697279    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.697366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697320    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697348    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.697554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697388    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.697554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697431    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.697554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697507    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.697713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697559    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.697713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697625    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697710    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.697817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697757    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697790    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.697985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697849    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.697985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697891    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.697985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697921    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.698140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.697982    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.698140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698081    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.698140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698091    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.698296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698225    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.698296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698277    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.698401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698352    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.698401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698390    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.698527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698433    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.698527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698514    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc\") pod \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\" (UID: \"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377\") "
Oct 31 00:27:45.698688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698558    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy\") pod \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\" (UID: \"bba615fc-186e-48cb-9bf1-18ee9c33dc9a\") "
Oct 31 00:27:45.698688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698596    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.698688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698672    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.698842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698732    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket\") pod \"690002f6-4096-42d6-bcf2-250b98c55207\" (UID: \"690002f6-4096-42d6-bcf2-250b98c55207\") "
Oct 31 00:27:45.698842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.698761    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket\") pod \"4da9275e-742e-4820-8580-c48195fe6776\" (UID: \"4da9275e-742e-4820-8580-c48195fe6776\") "
Oct 31 00:27:45.699912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.699890    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.700030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.699914    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.700030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.699928    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.701789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.700755    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.701789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.701011    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.701789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.701202    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.701789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.701454    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.701789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.701752    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.702980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.702940    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.706852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.706344    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.711105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.711074    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.711658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.711635    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.711874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.711854    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.712689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.712661    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:45.712804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.712784    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.713763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.713717    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.713924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.713903    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.714605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.714345    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:45.714605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.714454    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:45.714769 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.714738    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.714973    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "kube-api-access-9qvcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.715004    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.715181    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.715360    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.715418    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.715587    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.716235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.716123    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.717073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.716632    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.717073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.717059    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7" pod "4da9275e-742e-4820-8580-c48195fe6776" (UID: "4da9275e-742e-4820-8580-c48195fe6776"). InnerVolumeSpecName "kube-api-access-kwsp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.717560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.717528    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "kube-api-access-zl628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.717850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.717822    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.718163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.718133    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:45.718899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.718740    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:45.719920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.719887    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78" pod "690002f6-4096-42d6-bcf2-250b98c55207" (UID: "690002f6-4096-42d6-bcf2-250b98c55207"). InnerVolumeSpecName "kube-api-access-p4j78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:45.721679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.721628    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy" pod "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" (UID: "85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:45.722066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.722036    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo" pod "bba615fc-186e-48cb-9bf1-18ee9c33dc9a" (UID: "bba615fc-186e-48cb-9bf1-18ee9c33dc9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:45.801495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801449    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da9275e-742e-4820-8580-c48195fe6776-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801491    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801500    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801510    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da9275e-742e-4820-8580-c48195fe6776-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801519    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801528    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801537    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801547    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801556    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801564    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801572    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801581    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl628\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-kube-api-access-zl628\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801589    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801597    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801606    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801615    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690002f6-4096-42d6-bcf2-250b98c55207-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801623    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801631    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801640    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801649    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-kube-api-access-p4j78\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801658    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801668    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801677    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377-kube-api-access-9qvcc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801686    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801694    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690002f6-4096-42d6-bcf2-250b98c55207-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801703    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsp7\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-kube-api-access-kwsp7\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801713    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690002f6-4096-42d6-bcf2-250b98c55207-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801722    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da9275e-742e-4820-8580-c48195fe6776-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.801721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801732    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da9275e-742e-4820-8580-c48195fe6776-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.803013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801740    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.803013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801748    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690002f6-4096-42d6-bcf2-250b98c55207-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.803013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801756    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:45.803013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:45.801764    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba615fc-186e-48cb-9bf1-18ee9c33dc9a-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:46.209396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.208854    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:46.216452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.216426    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:46.466180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.466101    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"]
Oct 31 00:27:46.467081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467059    5078 cpu_manager.go:414] "containerMap: removing container" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy"
Oct 31 00:27:46.467081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467080    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy"
Oct 31 00:27:46.467300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467114    5078 cpu_manager.go:414] "containerMap: removing container" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy"
Oct 31 00:27:46.467300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467121    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy"
Oct 31 00:27:46.467300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467140    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy"
Oct 31 00:27:46.467300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467149    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy"
Oct 31 00:27:46.467300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467199    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy"
Oct 31 00:27:46.467300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467206    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy"
Oct 31 00:27:46.467677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467405    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da9275e-742e-4820-8580-c48195fe6776" containerName="istio-proxy"
Oct 31 00:27:46.467677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467428    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" containerName="istio-proxy"
Oct 31 00:27:46.467677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467442    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="690002f6-4096-42d6-bcf2-250b98c55207" containerName="istio-proxy"
Oct 31 00:27:46.467677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.467456    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" containerName="istio-proxy"
Oct 31 00:27:46.480995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.476938    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.480995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.480876    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"]
Oct 31 00:27:46.489319 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.487119    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d898f3-serving-cert\""
Oct 31 00:27:46.559232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.559191    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw" event={"ID":"85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377","Type":"ContainerDied","Data":"967f76e8003e5f8601bb1ab9938fca81aa8b43b8f58d1ec589d6f024f9ea7862"}
Oct 31 00:27:46.559755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.559264    5078 scope.go:122] "RemoveContainer" containerID="2eb948f8ec0d61796cf275d27a9c21e292c4ef3dd00beb8674c6aabcbfa4cc7d"
Oct 31 00:27:46.559755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.559414    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"
Oct 31 00:27:46.564599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.564580    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"
Oct 31 00:27:46.567076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.567059    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"
Oct 31 00:27:46.567712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.567175    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq" event={"ID":"bba615fc-186e-48cb-9bf1-18ee9c33dc9a","Type":"ContainerDied","Data":"7dea302e3457b0493d6e004620fd56fab7bb14bd746ac46f35f6d1eb1a99a2d4"}
Oct 31 00:27:46.569505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.569479    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7" event={"ID":"4da9275e-742e-4820-8580-c48195fe6776","Type":"ContainerDied","Data":"aff4cd8d1a77960882304c93701d82ca99195bc4fbfccaa3a46632473bf1f88c"}
Oct 31 00:27:46.569621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.569522    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"
Oct 31 00:27:46.570571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.570554    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:46.571767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.571748    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:27:46.582604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.582589    5078 scope.go:122] "RemoveContainer" containerID="7bbb4b7931d9b1edb29049aa378dd4e6dc96d69f7432afc53a7cfb7a810d25e1"
Oct 31 00:27:46.594401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.594385    5078 scope.go:122] "RemoveContainer" containerID="4b3945c907d84a1579b7b3f8fe3e8fb2a8e7914e79f98c3a5e385e9d768cf8f7"
Oct 31 00:27:46.619146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.619117    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"]
Oct 31 00:27:46.621370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.621337    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.621502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.621389    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.621594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.621542    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clb29\" (UniqueName: \"kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.621796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.621659    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.625274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.625254    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-cz6zq"]
Oct 31 00:27:46.692498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.686984    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"]
Oct 31 00:27:46.708545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.701295    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-d7zlw"]
Oct 31 00:27:46.708801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.708770    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"]
Oct 31 00:27:46.715523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.714767    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-8d8ft"]
Oct 31 00:27:46.728775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.728689    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.729029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.729011    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.729503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.729312    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.729622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.729253    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clb29\" (UniqueName: \"kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.729912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.729834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.736541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.732978    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.736672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.736562    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"]
Oct 31 00:27:46.737586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.737544    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.747543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.747516    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clb29\" (UniqueName: \"kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29\") pod \"maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:46.752411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.752378    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-mvvj7"]
Oct 31 00:27:46.807831 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:46.807798    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:47.401818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:47.401769    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"]
Oct 31 00:27:47.583766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:47.583671    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr" event={"ID":"381d0146-cbfd-4a00-bdff-7947f9f2a459","Type":"ContainerStarted","Data":"1e81822654ecedb1b79badb81015de46db6dd941d32dccefd4c1d62ee4193ae0"}
Oct 31 00:27:48.199697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.199662    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da9275e-742e-4820-8580-c48195fe6776" path="/var/lib/kubelet/pods/4da9275e-742e-4820-8580-c48195fe6776/volumes"
Oct 31 00:27:48.200197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.200182    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690002f6-4096-42d6-bcf2-250b98c55207" path="/var/lib/kubelet/pods/690002f6-4096-42d6-bcf2-250b98c55207/volumes"
Oct 31 00:27:48.200699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.200685    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377" path="/var/lib/kubelet/pods/85b7a6bf-9d4d-4c36-ae14-19a2bf4d0377/volumes"
Oct 31 00:27:48.201518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.201502    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba615fc-186e-48cb-9bf1-18ee9c33dc9a" path="/var/lib/kubelet/pods/bba615fc-186e-48cb-9bf1-18ee9c33dc9a/volumes"
Oct 31 00:27:48.604413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.604378    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr" event={"ID":"381d0146-cbfd-4a00-bdff-7947f9f2a459","Type":"ContainerStarted","Data":"d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7"}
Oct 31 00:27:48.604972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.604677    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:48.631494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:48.631422    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr" podStartSLOduration=2.073335456 podStartE2EDuration="2.631405612s" totalImagesPullingTime="558.070156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:48.629825392 +0000 UTC m=+2036.576196128" watchObservedRunningTime="2026-10-31 00:27:48.631405612 +0000 UTC m=+2036.577776329"
Oct 31 00:27:49.339494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.326012    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d" podUID="7b35096f-b88a-4ef8-8280-1ff7b12f2868" containerName="payload-processing" containerID="cri-o://ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958" gracePeriod=25
Oct 31 00:27:49.514609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.514574    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"]
Oct 31 00:27:49.576944 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.575855    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy" containerID="cri-o://71d1bd195037b7c88fa1334098ca04cdd7de630643d9b3101d54cc75ab1f7338" gracePeriod=30
Oct 31 00:27:49.603385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.585545    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"]
Oct 31 00:27:49.603385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.585786    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy" containerID="cri-o://bede096fa117f7f58ead2dc017c904d6e94345f71ed05806efbe7f23f682ed6b" gracePeriod=30
Oct 31 00:27:49.678371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.678333    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"]
Oct 31 00:27:49.678941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.678589    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" containerID="cri-o://1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08" gracePeriod=30
Oct 31 00:27:49.699314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.697221    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"]
Oct 31 00:27:49.699314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.697439    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" containerID="cri-o://a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e" gracePeriod=30
Oct 31 00:27:49.970957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.970923    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"]
Oct 31 00:27:49.976071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.976050    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:49.983648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.983158    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2d9875\""
Oct 31 00:27:49.995525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:49.995500    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"]
Oct 31 00:27:50.025791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.025759    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:27:50.026156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.026134    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"]
Oct 31 00:27:50.027943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.027918    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7b35096f-b88a-4ef8-8280-1ff7b12f2868" containerName="payload-processing"
Oct 31 00:27:50.027943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.027943    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b35096f-b88a-4ef8-8280-1ff7b12f2868" containerName="payload-processing"
Oct 31 00:27:50.028191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.028176    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b35096f-b88a-4ef8-8280-1ff7b12f2868" containerName="payload-processing"
Oct 31 00:27:50.037694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.037670    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"]
Oct 31 00:27:50.038363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.037781    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.050364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.050207    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2d9875-dockercfg-ctt5g\""
Oct 31 00:27:50.112640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.112599    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.112822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.112685    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4l5\" (UniqueName: \"kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5\") pod \"payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.112966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.112947    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxf9\" (UniqueName: \"kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9\") pod \"payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.113100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.112999    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.291483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.283679    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn\" (UniqueName: \"kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn\") pod \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") "
Oct 31 00:27:50.291483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.283972    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume\") pod \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\" (UID: \"7b35096f-b88a-4ef8-8280-1ff7b12f2868\") "
Oct 31 00:27:50.291483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.284936    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxf9\" (UniqueName: \"kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9\") pod \"payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.291483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.289625    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.291483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.289752    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.291483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.289791    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4l5\" (UniqueName: \"kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5\") pod \"payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.312607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.311440    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume" pod "7b35096f-b88a-4ef8-8280-1ff7b12f2868" (UID: "7b35096f-b88a-4ef8-8280-1ff7b12f2868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:50.312607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.312274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.314148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.313698    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.375505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.365156    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxf9\" (UniqueName: \"kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9\") pod \"payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.375505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.366873    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:50.390495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.385153    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4l5\" (UniqueName: \"kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5\") pod \"payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.392552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.392509    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b35096f-b88a-4ef8-8280-1ff7b12f2868-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:50.424591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.424517    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"]
Oct 31 00:27:50.534787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.532931    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08673d2c-ddgpf"]
Oct 31 00:27:50.562541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.562104    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn" pod "7b35096f-b88a-4ef8-8280-1ff7b12f2868" (UID: "7b35096f-b88a-4ef8-8280-1ff7b12f2868"). InnerVolumeSpecName "kube-api-access-gqvrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:50.637717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.637664    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvrn\" (UniqueName: \"kubernetes.io/projected/7b35096f-b88a-4ef8-8280-1ff7b12f2868-kube-api-access-gqvrn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:50.640646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.640392    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:50.666900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.666868    5078 generic.go:356] "Generic (PLEG): container finished" podID="7b35096f-b88a-4ef8-8280-1ff7b12f2868" containerID="ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958" exitCode=0
Oct 31 00:27:50.667043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.666980    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d" event={"ID":"7b35096f-b88a-4ef8-8280-1ff7b12f2868","Type":"ContainerDied","Data":"ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958"}
Oct 31 00:27:50.667043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.667021    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d" event={"ID":"7b35096f-b88a-4ef8-8280-1ff7b12f2868","Type":"ContainerDied","Data":"f3f0064abc0649499001d66479640726391849d8d4da525bd22631e83be0055d"}
Oct 31 00:27:50.667043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.667041    5078 scope.go:122] "RemoveContainer" containerID="ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958"
Oct 31 00:27:50.667215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.667161    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"
Oct 31 00:27:50.723954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.723165    5078 scope.go:122] "RemoveContainer" containerID="ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958"
Oct 31 00:27:50.724367 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:50.724089    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958\": container with ID starting with ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958 not found: ID does not exist" containerID="ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958"
Oct 31 00:27:50.724367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.724133    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958"} err="failed to get container status \"ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958\": rpc error: code = NotFound desc = could not find container \"ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958\": container with ID starting with ae36d5a69b6aca7e9776599c78927915395d8ecb1aa7fff49a825ac27c74e958 not found: ID does not exist"
Oct 31 00:27:50.757035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.757006    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"]
Oct 31 00:27:50.771374 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.771344    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-82r2d"]
Oct 31 00:27:50.857998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.857953    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"]
Oct 31 00:27:50.858737 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:27:50.858707    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc629a17_3d53_4c71_8ca9_8d140ee3618a.slice/crio-53353b3b94530bcc07d360bf396abd0fda9e1322e9a16e3bc52bb1c5b1789ee5 WatchSource:0}: Error finding container 53353b3b94530bcc07d360bf396abd0fda9e1322e9a16e3bc52bb1c5b1789ee5: Status 404 returned error can't find the container with id 53353b3b94530bcc07d360bf396abd0fda9e1322e9a16e3bc52bb1c5b1789ee5
Oct 31 00:27:50.970528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:50.970498    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"]
Oct 31 00:27:51.679965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.679927    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r" event={"ID":"0165dc8b-2e01-4dba-a5bc-04c847535b64","Type":"ContainerStarted","Data":"da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818"}
Oct 31 00:27:51.680174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.679988    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:27:51.680174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.680009    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r" event={"ID":"0165dc8b-2e01-4dba-a5bc-04c847535b64","Type":"ContainerStarted","Data":"869fd8495d89c0ab43cf240d52ea306784a21e671d7c23349d5882d8d5b4934a"}
Oct 31 00:27:51.683341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.683316    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q" event={"ID":"bc629a17-3d53-4c71-8ca9-8d140ee3618a","Type":"ContainerStarted","Data":"a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0"}
Oct 31 00:27:51.683523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.683371    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:27:51.683523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.683400    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q" event={"ID":"bc629a17-3d53-4c71-8ca9-8d140ee3618a","Type":"ContainerStarted","Data":"53353b3b94530bcc07d360bf396abd0fda9e1322e9a16e3bc52bb1c5b1789ee5"}
Oct 31 00:27:51.705292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.705240    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r" podStartSLOduration=2.705221678 podStartE2EDuration="2.705221678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:51.704344534 +0000 UTC m=+2039.650715259" watchObservedRunningTime="2026-10-31 00:27:51.705221678 +0000 UTC m=+2039.651592404"
Oct 31 00:27:51.735149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.735108    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q" podStartSLOduration=2.735092199 podStartE2EDuration="2.735092199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:51.731367142 +0000 UTC m=+2039.677737863" watchObservedRunningTime="2026-10-31 00:27:51.735092199 +0000 UTC m=+2039.681462915"
Oct 31 00:27:51.959161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.959077    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:27:51.959161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:51.959137    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:27:52.151966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:52.151926    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:27:52.152216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:52.151989    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:27:52.200816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:52.200784    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b35096f-b88a-4ef8-8280-1ff7b12f2868" path="/var/lib/kubelet/pods/7b35096f-b88a-4ef8-8280-1ff7b12f2868/volumes"
Oct 31 00:27:52.201291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:52.201277    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21ffa9f-c28d-45df-a0a5-4dc82f719693" path="/var/lib/kubelet/pods/f21ffa9f-c28d-45df-a0a5-4dc82f719693/volumes"
Oct 31 00:27:54.659646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:54.659613    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:27:54.726415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:54.726383    5078 generic.go:356] "Generic (PLEG): container finished" podID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerID="bede096fa117f7f58ead2dc017c904d6e94345f71ed05806efbe7f23f682ed6b" exitCode=0
Oct 31 00:27:54.726563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:54.726475    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" event={"ID":"48402b00-cb93-46fe-87c6-4e1437aca83c","Type":"ContainerDied","Data":"bede096fa117f7f58ead2dc017c904d6e94345f71ed05806efbe7f23f682ed6b"}
Oct 31 00:27:54.726563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:54.726553    5078 scope.go:122] "RemoveContainer" containerID="558a54efa4466a8231c82bbe4ba518f3e330770b35d8ae092de99be0843a513b"
Oct 31 00:27:54.730955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:54.730927    5078 generic.go:356] "Generic (PLEG): container finished" podID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerID="71d1bd195037b7c88fa1334098ca04cdd7de630643d9b3101d54cc75ab1f7338" exitCode=0
Oct 31 00:27:54.731279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:54.731258    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" event={"ID":"3ad9fb99-b725-4f80-b139-ceaba025da2c","Type":"ContainerDied","Data":"71d1bd195037b7c88fa1334098ca04cdd7de630643d9b3101d54cc75ab1f7338"}
Oct 31 00:27:55.037368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.037038    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:27:55.083852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.083825    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:27:55.183180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183130    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183201    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183249    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183272    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183333    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183433    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183515    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183562    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183584    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183667    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183692    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183720    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183745    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183785    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183820    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.183852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183850    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.184323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183883    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket\") pod \"3ad9fb99-b725-4f80-b139-ceaba025da2c\" (UID: \"3ad9fb99-b725-4f80-b139-ceaba025da2c\") "
Oct 31 00:27:55.184323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.183904    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data\") pod \"48402b00-cb93-46fe-87c6-4e1437aca83c\" (UID: \"48402b00-cb93-46fe-87c6-4e1437aca83c\") "
Oct 31 00:27:55.185834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.185806    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.186755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.186318    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.186755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.186347    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:55.186755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.186523    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.187423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.186951    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.187423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.186972    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.187423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.187284    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:55.190648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.190629    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.190883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.190872    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.194647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.192827    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.194647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.192928    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.194647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.193054    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.203154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.203132    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:27:55.212686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.212663    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:55.214675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.214652    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "kube-api-access-gqwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.215172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.215153    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:55.218005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.217959    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy" pod "3ad9fb99-b725-4f80-b139-ceaba025da2c" (UID: "3ad9fb99-b725-4f80-b139-ceaba025da2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.221211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.221189    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "kube-api-access-f4cbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.224894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.224860    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy" pod "48402b00-cb93-46fe-87c6-4e1437aca83c" (UID: "48402b00-cb93-46fe-87c6-4e1437aca83c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.249445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.249422    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:27:55.292380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.292342    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.292553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.292532    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.292793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.292767    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.292910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.292812    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.292910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.292854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.293033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.292945    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.293033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.293006    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.293126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.293040    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.293179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.293146    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.293231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.293199    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy\") pod \"a123a837-3763-4b5f-b18d-7281b1ebcc63\" (UID: \"a123a837-3763-4b5f-b18d-7281b1ebcc63\") "
Oct 31 00:27:55.293578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.293550    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:55.294303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294277    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294305    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48402b00-cb93-46fe-87c6-4e1437aca83c-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294320    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqwd2\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-kube-api-access-gqwd2\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294335    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294349    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294362    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294375    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294388    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294401    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294415    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4cbn\" (UniqueName: \"kubernetes.io/projected/48402b00-cb93-46fe-87c6-4e1437aca83c-kube-api-access-f4cbn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294428    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294442    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a123a837-3763-4b5f-b18d-7281b1ebcc63-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294456    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294549    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294564    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48402b00-cb93-46fe-87c6-4e1437aca83c-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294579    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad9fb99-b725-4f80-b139-ceaba025da2c-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294592    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad9fb99-b725-4f80-b139-ceaba025da2c-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294614    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294628    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.294820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.294641    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48402b00-cb93-46fe-87c6-4e1437aca83c-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.295439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.295366    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.299661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.299632    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.300675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.300648    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.300924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.300903    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.304944 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.304917    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "kube-api-access-kf672". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.305857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.305825    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.308272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.308248    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo" pod "a123a837-3763-4b5f-b18d-7281b1ebcc63" (UID: "a123a837-3763-4b5f-b18d-7281b1ebcc63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:55.395998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.395974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396018    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396037    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396061    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396392    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396484    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:55.396576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396536    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396573    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396683    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.396883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.396827    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.397116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.397073    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket\") pod \"54e1e46f-db63-441c-a3ce-161b24384674\" (UID: \"54e1e46f-db63-441c-a3ce-161b24384674\") "
Oct 31 00:27:55.398198 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398173    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e1e46f-db63-441c-a3ce-161b24384674-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398320    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398415    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398574    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398671 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398660    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398749    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf672\" (UniqueName: \"kubernetes.io/projected/a123a837-3763-4b5f-b18d-7281b1ebcc63-kube-api-access-kf672\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398845 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398833    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.398930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398918    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.399017 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.399007    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a123a837-3763-4b5f-b18d-7281b1ebcc63-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.399088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.398958    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.399169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.399115    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.399264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.399216    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:55.399264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.399240    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.399393 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.399347    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.399393 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.399375    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "kube-api-access-4nmf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:55.406539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.405519    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy" pod "54e1e46f-db63-441c-a3ce-161b24384674" (UID: "54e1e46f-db63-441c-a3ce-161b24384674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507660    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e1e46f-db63-441c-a3ce-161b24384674-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507697    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507721    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507737    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmf9\" (UniqueName: \"kubernetes.io/projected/54e1e46f-db63-441c-a3ce-161b24384674-kube-api-access-4nmf9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507751    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507765    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.507818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.507779    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e1e46f-db63-441c-a3ce-161b24384674-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:55.746014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.745912    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"
Oct 31 00:27:55.746014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.745938    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd" event={"ID":"3ad9fb99-b725-4f80-b139-ceaba025da2c","Type":"ContainerDied","Data":"5b721aefa8592dd83d269687c527113174a534c3141e53dd37279666eb0f22b2"}
Oct 31 00:27:55.746544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.746022    5078 scope.go:122] "RemoveContainer" containerID="71d1bd195037b7c88fa1334098ca04cdd7de630643d9b3101d54cc75ab1f7338"
Oct 31 00:27:55.748116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.748086    5078 generic.go:356] "Generic (PLEG): container finished" podID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerID="1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08" exitCode=0
Oct 31 00:27:55.748220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.748140    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"
Oct 31 00:27:55.748220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.748132    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" event={"ID":"a123a837-3763-4b5f-b18d-7281b1ebcc63","Type":"ContainerDied","Data":"1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08"}
Oct 31 00:27:55.748220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.748205    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs" event={"ID":"a123a837-3763-4b5f-b18d-7281b1ebcc63","Type":"ContainerDied","Data":"913ee7c128586603c32029942d58506f44fd1bf000461a09baf04ef283c2caa8"}
Oct 31 00:27:55.750018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.749995    5078 generic.go:356] "Generic (PLEG): container finished" podID="54e1e46f-db63-441c-a3ce-161b24384674" containerID="a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e" exitCode=0
Oct 31 00:27:55.750107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.750049    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"
Oct 31 00:27:55.750158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.750094    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" event={"ID":"54e1e46f-db63-441c-a3ce-161b24384674","Type":"ContainerDied","Data":"a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e"}
Oct 31 00:27:55.750239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.750162    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx" event={"ID":"54e1e46f-db63-441c-a3ce-161b24384674","Type":"ContainerDied","Data":"1b4625576ba0aa8e81c8fe17ec72990f04ac6386270ecc7c6cdba5eeb2e92bb5"}
Oct 31 00:27:55.751903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.751882    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr" event={"ID":"48402b00-cb93-46fe-87c6-4e1437aca83c","Type":"ContainerDied","Data":"142866db55d9d812f54c582c1544e4f27040541b10e194104209552827560600"}
Oct 31 00:27:55.752003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.751967    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"
Oct 31 00:27:55.765742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.765727    5078 scope.go:122] "RemoveContainer" containerID="1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08"
Oct 31 00:27:55.781824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.781800    5078 scope.go:122] "RemoveContainer" containerID="1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08"
Oct 31 00:27:55.782606 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:55.782117    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08\": container with ID starting with 1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08 not found: ID does not exist" containerID="1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08"
Oct 31 00:27:55.782606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.782153    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08"} err="failed to get container status \"1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08\": rpc error: code = NotFound desc = could not find container \"1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08\": container with ID starting with 1b72492dfa1a8da13612ef07378c2c0fb4bbaf76ab31ce76401ded19ec983c08 not found: ID does not exist"
Oct 31 00:27:55.782606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.782176    5078 scope.go:122] "RemoveContainer" containerID="a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e"
Oct 31 00:27:55.794696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.794675    5078 scope.go:122] "RemoveContainer" containerID="a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e"
Oct 31 00:27:55.795044 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:27:55.795019    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e\": container with ID starting with a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e not found: ID does not exist" containerID="a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e"
Oct 31 00:27:55.795130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.795057    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e"} err="failed to get container status \"a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e\": rpc error: code = NotFound desc = could not find container \"a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e\": container with ID starting with a9d2df8b3ed6a4261b165eed44897b910c38284bea660ba9e8d2d2b444947f6e not found: ID does not exist"
Oct 31 00:27:55.795130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.795080    5078 scope.go:122] "RemoveContainer" containerID="bede096fa117f7f58ead2dc017c904d6e94345f71ed05806efbe7f23f682ed6b"
Oct 31 00:27:55.800338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.800271    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"]
Oct 31 00:27:55.807632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.807584    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-gfhbs"]
Oct 31 00:27:55.824509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.824480    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"]
Oct 31 00:27:55.829363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.829342    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9j2kd"]
Oct 31 00:27:55.844512 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.844492    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"]
Oct 31 00:27:55.852201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.852178    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-gjsxr"]
Oct 31 00:27:55.867261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.867236    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"]
Oct 31 00:27:55.874772 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:55.874752    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-cnfpx"]
Oct 31 00:27:56.282380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:56.282349    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" path="/var/lib/kubelet/pods/3ad9fb99-b725-4f80-b139-ceaba025da2c/volumes"
Oct 31 00:27:56.289483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:56.283482    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" path="/var/lib/kubelet/pods/48402b00-cb93-46fe-87c6-4e1437aca83c/volumes"
Oct 31 00:27:56.307487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:56.292078    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e1e46f-db63-441c-a3ce-161b24384674" path="/var/lib/kubelet/pods/54e1e46f-db63-441c-a3ce-161b24384674/volumes"
Oct 31 00:27:56.309071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:56.309039    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" path="/var/lib/kubelet/pods/a123a837-3763-4b5f-b18d-7281b1ebcc63/volumes"
Oct 31 00:27:57.198201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.198167    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"]
Oct 31 00:27:57.207690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.204619    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy" containerID="cri-o://1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5" gracePeriod=30
Oct 31 00:27:57.245562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.241424    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"]
Oct 31 00:27:57.245562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.241633    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" podUID="b5c1b29b-5502-46d7-bade-e05d3be82fe0" containerName="maas-api" containerID="cri-o://d496dc1bbb184391d6fdb60dcb462e62f6da06022b8fc2df0494343cca725fe7" gracePeriod=30
Oct 31 00:27:57.952089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.952036    5078 generic.go:356] "Generic (PLEG): container finished" podID="b5c1b29b-5502-46d7-bade-e05d3be82fe0" containerID="d496dc1bbb184391d6fdb60dcb462e62f6da06022b8fc2df0494343cca725fe7" exitCode=0
Oct 31 00:27:57.952505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.952481    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" event={"ID":"b5c1b29b-5502-46d7-bade-e05d3be82fe0","Type":"ContainerDied","Data":"d496dc1bbb184391d6fdb60dcb462e62f6da06022b8fc2df0494343cca725fe7"}
Oct 31 00:27:57.970229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.970198    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:27:57.977681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.977186    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x\" (UniqueName: \"kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x\") pod \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") "
Oct 31 00:27:57.977681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.977292    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") pod \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") "
Oct 31 00:27:57.978049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.978027    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs\") pod \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") "
Oct 31 00:27:57.978314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.978293    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca\") pod \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\" (UID: \"b5c1b29b-5502-46d7-bade-e05d3be82fe0\") "
Oct 31 00:27:57.978986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.978956    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca" pod "b5c1b29b-5502-46d7-bade-e05d3be82fe0" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:57.980173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.980129    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x" pod "b5c1b29b-5502-46d7-bade-e05d3be82fe0" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0"). InnerVolumeSpecName "kube-api-access-9jl6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:57.982101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.982077    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1b29b-5502-46d7-bade-e05d3be82fe0-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.982208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.982103    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jl6x\" (UniqueName: \"kubernetes.io/projected/b5c1b29b-5502-46d7-bade-e05d3be82fe0-kube-api-access-9jl6x\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.982208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.982131    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs" pod "b5c1b29b-5502-46d7-bade-e05d3be82fe0" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:57.982477 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:57.982445    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls" pod "b5c1b29b-5502-46d7-bade-e05d3be82fe0" (UID: "b5c1b29b-5502-46d7-bade-e05d3be82fe0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:58.037340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.037305    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"]
Oct 31 00:27:58.042069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.042045    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-719e12-29v5p"]
Oct 31 00:27:58.085018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.084983    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:58.085018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.085016    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c1b29b-5502-46d7-bade-e05d3be82fe0-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:58.206537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.203885    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27" path="/var/lib/kubelet/pods/a94bba3f-f9b9-4d3f-8bc3-7fccc8658e27/volumes"
Oct 31 00:27:58.963583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.963549    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"
Oct 31 00:27:58.963758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.963590    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j" event={"ID":"b5c1b29b-5502-46d7-bade-e05d3be82fe0","Type":"ContainerDied","Data":"16e7aeb860f1f7934293e57fc68938f7da26f96d9f107c3d8d6212c4d1addf59"}
Oct 31 00:27:58.963758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.963664    5078 scope.go:122] "RemoveContainer" containerID="d496dc1bbb184391d6fdb60dcb462e62f6da06022b8fc2df0494343cca725fe7"
Oct 31 00:27:58.995586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:58.995560    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"]
Oct 31 00:27:59.003566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:27:59.003541    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-719e12-75bd5c569b-67g2j"]
Oct 31 00:28:00.200438 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:00.200391    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c1b29b-5502-46d7-bade-e05d3be82fe0" path="/var/lib/kubelet/pods/b5c1b29b-5502-46d7-bade-e05d3be82fe0/volumes"
Oct 31 00:28:02.552646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.552621    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:28:02.673240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673203    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673275    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673358    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673489    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673519    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673683    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673743    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.673851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673794    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:02.674216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.673904    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.674216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.674060    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:02.674216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.674096    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket\") pod \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\" (UID: \"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62\") "
Oct 31 00:28:02.674447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.674100    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:02.674447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.674169    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:02.674447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.674350    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:02.675025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.674997    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.675143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.675026    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.675143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.675042    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.675143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.675054    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.675143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.675067    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.676676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.676650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:02.677022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.676999    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "kube-api-access-c8n74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:02.677192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.677174    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:02.677283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.677256    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo" pod "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" (UID: "9b5a8ba3-79b1-4c91-a6f2-6e45de362f62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:28:02.718255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.716200    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:28:02.718255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.716619    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:28:02.778118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.778040    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8n74\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-kube-api-access-c8n74\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.778118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.778065    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.778118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.778076    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.778118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.778085    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:02.877127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.877093    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"]
Oct 31 00:28:02.878169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878112    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy"
Oct 31 00:28:02.878169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878135    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy"
Oct 31 00:28:02.878169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878169    5078 cpu_manager.go:414] "containerMap: removing container" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy"
Oct 31 00:28:02.878169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878173    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878194    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b5c1b29b-5502-46d7-bade-e05d3be82fe0" containerName="maas-api"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878198    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c1b29b-5502-46d7-bade-e05d3be82fe0" containerName="maas-api"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878211    5078 cpu_manager.go:414] "containerMap: removing container" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878216    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878225    5078 cpu_manager.go:414] "containerMap: removing container" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878229    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878248    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878252    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878258    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878264    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878402    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c1b29b-5502-46d7-bade-e05d3be82fe0" containerName="maas-api"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878415    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e1e46f-db63-441c-a3ce-161b24384674" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878422    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878430    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad9fb99-b725-4f80-b139-ceaba025da2c" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878436    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerName="istio-proxy"
Oct 31 00:28:02.878486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.878444    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a123a837-3763-4b5f-b18d-7281b1ebcc63" containerName="istio-proxy"
Oct 31 00:28:02.885347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.885327    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:02.914392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.913613    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0d4ad9-openshift-default-dockercfg-9mht6\""
Oct 31 00:28:02.920447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:02.920424    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"]
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024094    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024152    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024224    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024270    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024380    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024477    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024550    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024582    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.029818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.024714    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.167388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.139734    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.167699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.167668    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.167878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.167860    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.168061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.168045    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.168244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.168228    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.168399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.168385    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.168584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.168564    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.168730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.168715    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.168976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.168956    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.177811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.177769    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.178478 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.178436    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.189850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.189576    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.199661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.196608    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.199661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.197895    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.199661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.198428    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.213510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.210829    5078 generic.go:356] "Generic (PLEG): container finished" podID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" containerID="1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5" exitCode=0
Oct 31 00:28:03.213510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.211023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" event={"ID":"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62","Type":"ContainerDied","Data":"1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5"}
Oct 31 00:28:03.213510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.211087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8" event={"ID":"9b5a8ba3-79b1-4c91-a6f2-6e45de362f62","Type":"ContainerDied","Data":"b79386a1f1e770dd7589607df087967c455bde6e0434e7843c32608438f1d0e4"}
Oct 31 00:28:03.213510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.211118    5078 scope.go:122] "RemoveContainer" containerID="1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5"
Oct 31 00:28:03.213510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.211280    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"
Oct 31 00:28:03.278942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.220990    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.350409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.350375    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.371065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.354215    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48\") pod \"e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") " pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.386867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.386840    5078 scope.go:122] "RemoveContainer" containerID="1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5"
Oct 31 00:28:03.387208 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:28:03.387184    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5\": container with ID starting with 1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5 not found: ID does not exist" containerID="1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5"
Oct 31 00:28:03.387321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.387219    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5"} err="failed to get container status \"1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5\": rpc error: code = NotFound desc = could not find container \"1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5\": container with ID starting with 1c8e1dc6bd5d2522ecfd991517756994be4dcc635300f76c79393d837d29b6c5 not found: ID does not exist"
Oct 31 00:28:03.398561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.398524    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"]
Oct 31 00:28:03.405908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.405871    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-6mrn8"]
Oct 31 00:28:03.516134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.516101    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:03.833798 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:28:03.833760    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6de8b3_0e20_48c9_8843_cf574e833566.slice/crio-d06aa93c3cab5f75eadde89472fcd75d92a8ce6c1db396b895382c07984e2a91 WatchSource:0}: Error finding container d06aa93c3cab5f75eadde89472fcd75d92a8ce6c1db396b895382c07984e2a91: Status 404 returned error can't find the container with id d06aa93c3cab5f75eadde89472fcd75d92a8ce6c1db396b895382c07984e2a91
Oct 31 00:28:03.836445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.836418    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"]
Oct 31 00:28:03.837332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.837266    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:28:03.837444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.837345    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:28:03.837444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:03.837388    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:28:04.226720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:04.226670    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5a8ba3-79b1-4c91-a6f2-6e45de362f62" path="/var/lib/kubelet/pods/9b5a8ba3-79b1-4c91-a6f2-6e45de362f62/volumes"
Oct 31 00:28:04.231743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:04.231709    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" event={"ID":"9f6de8b3-0e20-48c9-8843-cf574e833566","Type":"ContainerStarted","Data":"a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"}
Oct 31 00:28:04.232589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:04.231765    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" event={"ID":"9f6de8b3-0e20-48c9-8843-cf574e833566","Type":"ContainerStarted","Data":"d06aa93c3cab5f75eadde89472fcd75d92a8ce6c1db396b895382c07984e2a91"}
Oct 31 00:28:04.259618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:04.259434    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podStartSLOduration=2.259415251 podStartE2EDuration="2.259415251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:04.258059021 +0000 UTC m=+2052.204429740" watchObservedRunningTime="2026-10-31 00:28:04.259415251 +0000 UTC m=+2052.205785973"
Oct 31 00:28:04.516653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:04.516565    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:04.523824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:04.523795    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:05.246784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:05.246618    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:05.247829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:05.247809    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:28:06.654216 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:28:06.654178    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:28:08.100500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.098974    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"]
Oct 31 00:28:08.100500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.099292    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" containerID="cri-o://2a86a0c197d966969edd953a281bb3f52a16de6b30bfc2b2305cf010b85bd695" gracePeriod=30
Oct 31 00:28:08.130974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.130941    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"]
Oct 31 00:28:08.131208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.131173    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy" containerID="cri-o://220dc17a9beadfbc1b81321b4b577cc377de699f94ea0e91f86f2609cc67a5f2" gracePeriod=30
Oct 31 00:28:08.606418 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.605346    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"]
Oct 31 00:28:08.608536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.607345    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="48402b00-cb93-46fe-87c6-4e1437aca83c" containerName="istio-proxy"
Oct 31 00:28:08.629621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.629593    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:08.707299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.706375    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:08.707299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.706613    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6z4\" (UniqueName: \"kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4\") pod \"payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:08.718306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.717568    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0d4ad9-dockercfg-frtzx\""
Oct 31 00:28:08.743062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.743015    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0d4ad9\""
Oct 31 00:28:08.755235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.755197    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"]
Oct 31 00:28:08.755504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.755489    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"]
Oct 31 00:28:08.828698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.813759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:08.828698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.813967    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6z4\" (UniqueName: \"kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4\") pod \"payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:08.828698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.820127    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:08.830982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.830413    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"]
Oct 31 00:28:08.830982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.830560    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:08.933489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.921710    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"]
Oct 31 00:28:08.933489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:08.922143    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6z4\" (UniqueName: \"kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4\") pod \"payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:09.021485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.013690    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"]
Oct 31 00:28:09.021485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.013857    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.046498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.022768    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:28:09.046498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.026428    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:28:09.054315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.054252    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.054685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.054654    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.054942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.054922    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.055195 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.055176    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbnn\" (UniqueName: \"kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn\") pod \"payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.057388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.055356    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5gt\" (UniqueName: \"kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.155719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.155676    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:09.159300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.159220    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.159379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.159331    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbnn\" (UniqueName: \"kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn\") pod \"payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.159446 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.159384    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5gt\" (UniqueName: \"kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.159602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.159583    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.159680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.159648    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.160159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.160130    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.160451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.160432    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.162199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.162175    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.184502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.184398    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbnn\" (UniqueName: \"kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn\") pod \"payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.184502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.184406    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5gt\" (UniqueName: \"kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt\") pod \"maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.187768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.187559    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:09.388697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.387245    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:09.476670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.476051    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"]
Oct 31 00:28:09.535844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.535819    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"]
Oct 31 00:28:09.543031 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:28:09.542999    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e0a354_0a16_4829_a1b5_b2528190b54e.slice/crio-96dac526a761e9981394110ac6822624cf360dc24e06e18e397124b90ff93a82 WatchSource:0}: Error finding container 96dac526a761e9981394110ac6822624cf360dc24e06e18e397124b90ff93a82: Status 404 returned error can't find the container with id 96dac526a761e9981394110ac6822624cf360dc24e06e18e397124b90ff93a82
Oct 31 00:28:09.609611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.609576    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv" event={"ID":"97e0a354-0a16-4829-a1b5-b2528190b54e","Type":"ContainerStarted","Data":"96dac526a761e9981394110ac6822624cf360dc24e06e18e397124b90ff93a82"}
Oct 31 00:28:09.612207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.612179    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx" event={"ID":"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3","Type":"ContainerStarted","Data":"f67e428899b331b0044fb3142ac92cec91caa829fc0b78160ae66000fceb664f"}
Oct 31 00:28:09.709358 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:28:09.709331    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30e5dd1_5053_423e_859e_2770c7197f38.slice/crio-af9c67de87ff5b539846474dfb97f165a1da57c9eba6845dd2386ec0b642e0b0 WatchSource:0}: Error finding container af9c67de87ff5b539846474dfb97f165a1da57c9eba6845dd2386ec0b642e0b0: Status 404 returned error can't find the container with id af9c67de87ff5b539846474dfb97f165a1da57c9eba6845dd2386ec0b642e0b0
Oct 31 00:28:09.713427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.713357    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"]
Oct 31 00:28:09.958431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.958397    5078 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:28:09.958621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:09.958506    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:28:10.027694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.027656    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"]
Oct 31 00:28:10.032292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.032264    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af86ada3-z6glk"]
Oct 31 00:28:10.201884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.201849    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e5fd92-5df1-44cc-a210-1e1534d35281" path="/var/lib/kubelet/pods/c7e5fd92-5df1-44cc-a210-1e1534d35281/volumes"
Oct 31 00:28:10.627976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.627924    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v" event={"ID":"d30e5dd1-5053-423e-859e-2770c7197f38","Type":"ContainerStarted","Data":"f02dbb66a2c11d4f2e6245ffd23f46c3fd4bfd619e82bac7e1af6961aa065978"}
Oct 31 00:28:10.628150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.627990    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v" event={"ID":"d30e5dd1-5053-423e-859e-2770c7197f38","Type":"ContainerStarted","Data":"af9c67de87ff5b539846474dfb97f165a1da57c9eba6845dd2386ec0b642e0b0"}
Oct 31 00:28:10.635199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.635168    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx" event={"ID":"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3","Type":"ContainerStarted","Data":"904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687"}
Oct 31 00:28:10.635853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.635833    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:10.638327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.638303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv" event={"ID":"97e0a354-0a16-4829-a1b5-b2528190b54e","Type":"ContainerStarted","Data":"1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0"}
Oct 31 00:28:10.638487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.638441    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:10.653837 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.653791    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v" podStartSLOduration=2.653778391 podStartE2EDuration="2.653778391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:10.651359326 +0000 UTC m=+2058.597730046" watchObservedRunningTime="2026-10-31 00:28:10.653778391 +0000 UTC m=+2058.600149108"
Oct 31 00:28:10.682419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.681857    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv" podStartSLOduration=2.68183604 podStartE2EDuration="2.68183604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:10.681289647 +0000 UTC m=+2058.627660365" watchObservedRunningTime="2026-10-31 00:28:10.68183604 +0000 UTC m=+2058.628206759"
Oct 31 00:28:10.757488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:10.749378    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx" podStartSLOduration=2.749355763 podStartE2EDuration="2.749355763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:10.713286118 +0000 UTC m=+2058.659656888" watchObservedRunningTime="2026-10-31 00:28:10.749355763 +0000 UTC m=+2058.695726480"
Oct 31 00:28:11.129911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.129883    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"]
Oct 31 00:28:11.137759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.137737    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.147835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.147668    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2d9875-serving-cert\""
Oct 31 00:28:11.152122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.151626    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"]
Oct 31 00:28:11.243850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.243809    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.244362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.243857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvqjm\" (UniqueName: \"kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.244362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.244159    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.244362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.244336    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.346079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.346042    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.346079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.346088    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqjm\" (UniqueName: \"kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.346605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.346580    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.346772 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.346723    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.347563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.347534    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.356262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.356016    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.356262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.356029    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.361415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.361390    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqjm\" (UniqueName: \"kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm\") pod \"maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.468683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.468587    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:11.652785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.652747    5078 generic.go:356] "Generic (PLEG): container finished" podID="d30e5dd1-5053-423e-859e-2770c7197f38" containerID="f02dbb66a2c11d4f2e6245ffd23f46c3fd4bfd619e82bac7e1af6961aa065978" exitCode=0
Oct 31 00:28:11.653708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.653659    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v" event={"ID":"d30e5dd1-5053-423e-859e-2770c7197f38","Type":"ContainerDied","Data":"f02dbb66a2c11d4f2e6245ffd23f46c3fd4bfd619e82bac7e1af6961aa065978"}
Oct 31 00:28:11.793660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:11.793264    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"]
Oct 31 00:28:12.665548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:12.665510    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2" event={"ID":"c10674b7-2ea5-435a-8bdc-a0a706dc351a","Type":"ContainerStarted","Data":"6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1"}
Oct 31 00:28:12.665983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:12.665578    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2" event={"ID":"c10674b7-2ea5-435a-8bdc-a0a706dc351a","Type":"ContainerStarted","Data":"532ccf4b977982f11cd87e99e33c1c3324f1db72ed9f10beddd9aff035841341"}
Oct 31 00:28:12.665983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:12.665610    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:12.717744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:12.717210    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2" podStartSLOduration=1.205658819 podStartE2EDuration="1.717184315s" totalImagesPullingTime="511.525496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:12.691177253 +0000 UTC m=+2060.637547971" watchObservedRunningTime="2026-10-31 00:28:12.717184315 +0000 UTC m=+2060.663555068"
Oct 31 00:28:13.317867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.317773    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:13.362199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.359065    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca\") pod \"d30e5dd1-5053-423e-859e-2770c7197f38\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") "
Oct 31 00:28:13.362199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.359200    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt\" (UniqueName: \"kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt\") pod \"d30e5dd1-5053-423e-859e-2770c7197f38\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") "
Oct 31 00:28:13.362199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.359291    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca\") pod \"d30e5dd1-5053-423e-859e-2770c7197f38\" (UID: \"d30e5dd1-5053-423e-859e-2770c7197f38\") "
Oct 31 00:28:13.362199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.360475    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca" pod "d30e5dd1-5053-423e-859e-2770c7197f38" (UID: "d30e5dd1-5053-423e-859e-2770c7197f38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:13.362199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.360660    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.408489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.368924    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca" pod "d30e5dd1-5053-423e-859e-2770c7197f38" (UID: "d30e5dd1-5053-423e-859e-2770c7197f38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:13.421327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.417500    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt" pod "d30e5dd1-5053-423e-859e-2770c7197f38" (UID: "d30e5dd1-5053-423e-859e-2770c7197f38"). InnerVolumeSpecName "kube-api-access-px5gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:13.466493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.466440    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30e5dd1-5053-423e-859e-2770c7197f38-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.466694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.466500    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5gt\" (UniqueName: \"kubernetes.io/projected/d30e5dd1-5053-423e-859e-2770c7197f38-kube-api-access-px5gt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.857850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.857808    5078 generic.go:356] "Generic (PLEG): container finished" podID="790147f1-a37a-4923-9375-9776c7635605" containerID="220dc17a9beadfbc1b81321b4b577cc377de699f94ea0e91f86f2609cc67a5f2" exitCode=0
Oct 31 00:28:13.858337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.858075    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" event={"ID":"790147f1-a37a-4923-9375-9776c7635605","Type":"ContainerDied","Data":"220dc17a9beadfbc1b81321b4b577cc377de699f94ea0e91f86f2609cc67a5f2"}
Oct 31 00:28:13.858337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.858132    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w" event={"ID":"790147f1-a37a-4923-9375-9776c7635605","Type":"ContainerDied","Data":"e1474f2a9828e529b19089e88177d41013766be940a6cc7c4769a909f90a817b"}
Oct 31 00:28:13.858337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.858155    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1474f2a9828e529b19089e88177d41013766be940a6cc7c4769a909f90a817b"
Oct 31 00:28:13.867533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.867506    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v" event={"ID":"d30e5dd1-5053-423e-859e-2770c7197f38","Type":"ContainerDied","Data":"af9c67de87ff5b539846474dfb97f165a1da57c9eba6845dd2386ec0b642e0b0"}
Oct 31 00:28:13.867700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.867550    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9c67de87ff5b539846474dfb97f165a1da57c9eba6845dd2386ec0b642e0b0"
Oct 31 00:28:13.867700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.867599    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"
Oct 31 00:28:13.868235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.868217    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:28:13.930352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.930228    5078 generic.go:356] "Generic (PLEG): container finished" podID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerID="2a86a0c197d966969edd953a281bb3f52a16de6b30bfc2b2305cf010b85bd695" exitCode=0
Oct 31 00:28:13.931166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.931130    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" event={"ID":"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7","Type":"ContainerDied","Data":"2a86a0c197d966969edd953a281bb3f52a16de6b30bfc2b2305cf010b85bd695"}
Oct 31 00:28:13.931321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:13.931202    5078 scope.go:122] "RemoveContainer" containerID="19cb826b1ec78cdb5567f5c750c2b4a34271a20f8325ad8baab9558b3bbc6714"
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.027845    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.028061    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.028127    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029216    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029274    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029424    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029503    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.029543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029552    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.030130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029696    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.030130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.029885    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket\") pod \"790147f1-a37a-4923-9375-9776c7635605\" (UID: \"790147f1-a37a-4923-9375-9776c7635605\") "
Oct 31 00:28:14.030279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.030256    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.031659    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.031687    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790147f1-a37a-4923-9375-9776c7635605-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.032890    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.033186    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.033266    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.033312    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.034723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.033493    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "kube-api-access-n4jhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:14.035095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.034823    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:28:14.035295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.035272    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token" pod "790147f1-a37a-4923-9375-9776c7635605" (UID: "790147f1-a37a-4923-9375-9776c7635605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:14.134003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.133947    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.134003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.133981    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790147f1-a37a-4923-9375-9776c7635605-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.134003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.134005    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.134297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.134020    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.134297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.134033    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790147f1-a37a-4923-9375-9776c7635605-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.134297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.134047    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jhn\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-kube-api-access-n4jhn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.134297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.134061    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790147f1-a37a-4923-9375-9776c7635605-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.141126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.141098    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:28:14.234656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234542    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234668    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234739    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234789    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234896    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234929    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.234969    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.235153    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.235253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket\") pod \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\" (UID: \"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7\") "
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.236022    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.236251    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.236320    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.236606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.236438    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.237349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.237004    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.238292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.238264    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "kube-api-access-5bpm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:14.238835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.238809    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:14.239723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.239684    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:14.239800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.239714    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo" pod "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" (UID: "d76dfb3f-5f97-4a12-af7f-56abaa6c24f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:28:14.337852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337816    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.337852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337850    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337868    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpm7\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-kube-api-access-5bpm7\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337883    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337898    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337913    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337927    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337942    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.338135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.337956    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:14.943363 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.943334    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"
Oct 31 00:28:14.943867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.943630    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"
Oct 31 00:28:14.944003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.943975    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g" event={"ID":"d76dfb3f-5f97-4a12-af7f-56abaa6c24f7","Type":"ContainerDied","Data":"30d28d3f69a79851fd610dcbb54c09c0d100d15be1666cd1ec3b50b8a48a349c"}
Oct 31 00:28:14.944157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.944141    5078 scope.go:122] "RemoveContainer" containerID="2a86a0c197d966969edd953a281bb3f52a16de6b30bfc2b2305cf010b85bd695"
Oct 31 00:28:14.968437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:14.968387    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"]
Oct 31 00:28:15.017289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:15.017254    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-7787w"]
Oct 31 00:28:15.023424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:15.023368    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"]
Oct 31 00:28:15.030829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:15.030802    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-vj72g"]
Oct 31 00:28:16.216582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.216546    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790147f1-a37a-4923-9375-9776c7635605" path="/var/lib/kubelet/pods/790147f1-a37a-4923-9375-9776c7635605/volumes"
Oct 31 00:28:16.217554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.217530    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" path="/var/lib/kubelet/pods/d76dfb3f-5f97-4a12-af7f-56abaa6c24f7/volumes"
Oct 31 00:28:16.929850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.929801    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"]
Oct 31 00:28:16.931001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.930967    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy"
Oct 31 00:28:16.931126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931116    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy"
Oct 31 00:28:16.931215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931206    5078 cpu_manager.go:414] "containerMap: removing container" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy"
Oct 31 00:28:16.931261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931255    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy"
Oct 31 00:28:16.931313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931307    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d30e5dd1-5053-423e-859e-2770c7197f38" containerName="revoke-keys"
Oct 31 00:28:16.931359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931353    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30e5dd1-5053-423e-859e-2770c7197f38" containerName="revoke-keys"
Oct 31 00:28:16.931622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931602    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy"
Oct 31 00:28:16.931622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931623    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="790147f1-a37a-4923-9375-9776c7635605" containerName="istio-proxy"
Oct 31 00:28:16.931832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.931631    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30e5dd1-5053-423e-859e-2770c7197f38" containerName="revoke-keys"
Oct 31 00:28:16.945933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.945906    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"]
Oct 31 00:28:16.946087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.946034    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:16.959846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.959823    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:28:16.961975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:16.961936    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:28:17.029611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.028834    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcpz4\" (UniqueName: \"kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.029611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.029054    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.029611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.029146    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.130950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.130911    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcpz4\" (UniqueName: \"kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.131157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.131096    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.131232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.131168    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.131860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.131832    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.133720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.133701    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.165479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.165444    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcpz4\" (UniqueName: \"kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4\") pod \"maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.342671 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.342575    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:17.715426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:17.715401    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"]
Oct 31 00:28:17.717642 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:28:17.717611    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65495e5a_7cfc_4f3a_841c_c4df05c2a8d7.slice/crio-8bfe97a6939d62fd86d4988a3c24982f27faf2be1edebe1cc13faad48a19b5a9 WatchSource:0}: Error finding container 8bfe97a6939d62fd86d4988a3c24982f27faf2be1edebe1cc13faad48a19b5a9: Status 404 returned error can't find the container with id 8bfe97a6939d62fd86d4988a3c24982f27faf2be1edebe1cc13faad48a19b5a9
Oct 31 00:28:18.047140 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:18.047040    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp" event={"ID":"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7","Type":"ContainerStarted","Data":"8bfe97a6939d62fd86d4988a3c24982f27faf2be1edebe1cc13faad48a19b5a9"}
Oct 31 00:28:18.938770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:18.938646    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:28:18.942004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:18.941968    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"]
Oct 31 00:28:19.093087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:19.093048    5078 generic.go:356] "Generic (PLEG): container finished" podID="65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" containerID="df383ec9849a853e257de5780b5a4a10b2eb546ec85ac176a76bb15c54a2c1ca" exitCode=0
Oct 31 00:28:19.093289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:19.093233    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp" event={"ID":"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7","Type":"ContainerDied","Data":"df383ec9849a853e257de5780b5a4a10b2eb546ec85ac176a76bb15c54a2c1ca"}
Oct 31 00:28:20.358947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.358853    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:20.424268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.424225    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca\") pod \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") "
Oct 31 00:28:20.424537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.424505    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4\" (UniqueName: \"kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4\") pod \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") "
Oct 31 00:28:20.424781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.424760    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca\") pod \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\" (UID: \"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7\") "
Oct 31 00:28:20.425978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.425949    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca" pod "65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" (UID: "65495e5a-7cfc-4f3a-841c-c4df05c2a8d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:20.427777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.427733    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4" pod "65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" (UID: "65495e5a-7cfc-4f3a-841c-c4df05c2a8d7"). InnerVolumeSpecName "kube-api-access-vcpz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:20.429423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.429395    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca" pod "65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" (UID: "65495e5a-7cfc-4f3a-841c-c4df05c2a8d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:20.527288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.527250    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcpz4\" (UniqueName: \"kubernetes.io/projected/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-kube-api-access-vcpz4\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:20.527288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.527282    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:20.527288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:20.527294    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:21.129713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:21.129681    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"
Oct 31 00:28:21.129713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:21.129700    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp" event={"ID":"65495e5a-7cfc-4f3a-841c-c4df05c2a8d7","Type":"ContainerDied","Data":"8bfe97a6939d62fd86d4988a3c24982f27faf2be1edebe1cc13faad48a19b5a9"}
Oct 31 00:28:21.130027 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:21.129782    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bfe97a6939d62fd86d4988a3c24982f27faf2be1edebe1cc13faad48a19b5a9"
Oct 31 00:28:21.657628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:21.657600    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:28:21.658096 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:21.657683    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:28:22.894728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.894693    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"]
Oct 31 00:28:22.895985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.895964    5078 cpu_manager.go:414] "containerMap: removing container" podUID="65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" containerName="revoke-keys"
Oct 31 00:28:22.895985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.895985    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" containerName="revoke-keys"
Oct 31 00:28:22.896194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.896005    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy"
Oct 31 00:28:22.896194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.896013    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy"
Oct 31 00:28:22.896840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.896282    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" containerName="revoke-keys"
Oct 31 00:28:22.896840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.896295    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76dfb3f-5f97-4a12-af7f-56abaa6c24f7" containerName="istio-proxy"
Oct 31 00:28:22.902650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.902625    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:22.907967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.906575    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"]
Oct 31 00:28:22.908991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.908917    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:28:22.912057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.912009    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:28:22.973109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.973029    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshf8\" (UniqueName: \"kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:22.973109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.973104    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:22.973332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:22.973139    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.075131 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.075094    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshf8\" (UniqueName: \"kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.075331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.075172    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.075331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.075210    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.075951 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.075922    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.078112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.078086    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.089109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.089087    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshf8\" (UniqueName: \"kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.152226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.152195    5078 generic.go:356] "Generic (PLEG): container finished" podID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerID="1a630d9284e37575cd1158c5a5edef094977a44e1abe5ed667cc27f72e85717a" exitCode=0
Oct 31 00:28:23.152386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.152269    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" event={"ID":"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da","Type":"ContainerDied","Data":"1a630d9284e37575cd1158c5a5edef094977a44e1abe5ed667cc27f72e85717a"}
Oct 31 00:28:23.152783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.152768    5078 scope.go:122] "RemoveContainer" containerID="1a630d9284e37575cd1158c5a5edef094977a44e1abe5ed667cc27f72e85717a"
Oct 31 00:28:23.154713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.154688    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:28:23.154794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.154757    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:28:23.154794 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.154787    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:28:23.225908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.225821    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:23.558175 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:28:23.557996    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ef8c95_01d1_4f94_8f33_efa06bdda094.slice/crio-3e6431ad71c4533436d93a9ef26906f8ebae2234a07dd61d2d763ed084d3bc19 WatchSource:0}: Error finding container 3e6431ad71c4533436d93a9ef26906f8ebae2234a07dd61d2d763ed084d3bc19: Status 404 returned error can't find the container with id 3e6431ad71c4533436d93a9ef26906f8ebae2234a07dd61d2d763ed084d3bc19
Oct 31 00:28:23.560203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.560163    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"]
Oct 31 00:28:23.942296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:23.942244    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm" podUID="cabd2b76-1571-4b37-85bc-5588b382f10f" containerName="payload-processing" containerID="cri-o://003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254" gracePeriod=25
Oct 31 00:28:24.025626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.025594    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:28:24.117504 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:28:24.117451    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabd2b76_1571_4b37_85bc_5588b382f10f.slice/crio-003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabd2b76_1571_4b37_85bc_5588b382f10f.slice/crio-conmon-003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:28:24.167930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.167890    5078 generic.go:356] "Generic (PLEG): container finished" podID="cabd2b76-1571-4b37-85bc-5588b382f10f" containerID="003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254" exitCode=0
Oct 31 00:28:24.168168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.167997    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm" event={"ID":"cabd2b76-1571-4b37-85bc-5588b382f10f","Type":"ContainerDied","Data":"003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254"}
Oct 31 00:28:24.191499 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.185112    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" event={"ID":"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da","Type":"ContainerStarted","Data":"15af77c0663c6c36392cd704dd7dec8a5dbc31d75b5973e50558bffa4c36350d"}
Oct 31 00:28:24.244033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.237622    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg" event={"ID":"41ef8c95-01d1-4f94-8f33-efa06bdda094","Type":"ContainerStarted","Data":"54d802f2eec8497b1e5385bd52737ff42b39afdaefb7165ea6361c4a46b2f028"}
Oct 31 00:28:24.244033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.237688    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg" event={"ID":"41ef8c95-01d1-4f94-8f33-efa06bdda094","Type":"ContainerStarted","Data":"3e6431ad71c4533436d93a9ef26906f8ebae2234a07dd61d2d763ed084d3bc19"}
Oct 31 00:28:24.308709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.308670    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:28:24.308952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.308778    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:28:24.308952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.308832    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:28:24.450065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.449695    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:28:24.525487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.519789    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg" podStartSLOduration=2.519768582 podStartE2EDuration="2.519768582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:24.26534672 +0000 UTC m=+2072.211717437" watchObservedRunningTime="2026-10-31 00:28:24.519768582 +0000 UTC m=+2072.466139300"
Oct 31 00:28:24.525487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.521813    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume\") pod \"cabd2b76-1571-4b37-85bc-5588b382f10f\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") "
Oct 31 00:28:24.525487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.522399    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume" pod "cabd2b76-1571-4b37-85bc-5588b382f10f" (UID: "cabd2b76-1571-4b37-85bc-5588b382f10f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:24.525487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.522584    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7\" (UniqueName: \"kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7\") pod \"cabd2b76-1571-4b37-85bc-5588b382f10f\" (UID: \"cabd2b76-1571-4b37-85bc-5588b382f10f\") "
Oct 31 00:28:24.525487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.524279    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabd2b76-1571-4b37-85bc-5588b382f10f-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:24.538308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.534262    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7" pod "cabd2b76-1571-4b37-85bc-5588b382f10f" (UID: "cabd2b76-1571-4b37-85bc-5588b382f10f"). InnerVolumeSpecName "kube-api-access-96lw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:24.627247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:24.627196    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lw7\" (UniqueName: \"kubernetes.io/projected/cabd2b76-1571-4b37-85bc-5588b382f10f-kube-api-access-96lw7\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:25.227405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.227372    5078 generic.go:356] "Generic (PLEG): container finished" podID="41ef8c95-01d1-4f94-8f33-efa06bdda094" containerID="54d802f2eec8497b1e5385bd52737ff42b39afdaefb7165ea6361c4a46b2f028" exitCode=0
Oct 31 00:28:25.227886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.227452    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg" event={"ID":"41ef8c95-01d1-4f94-8f33-efa06bdda094","Type":"ContainerDied","Data":"54d802f2eec8497b1e5385bd52737ff42b39afdaefb7165ea6361c4a46b2f028"}
Oct 31 00:28:25.229116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.229099    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"
Oct 31 00:28:25.229245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.229141    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm" event={"ID":"cabd2b76-1571-4b37-85bc-5588b382f10f","Type":"ContainerDied","Data":"55d5e0214b40ce054f6aa2dd3e86fd48c7cd4e4bb6a867fea781fefb955e7037"}
Oct 31 00:28:25.229245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.229183    5078 scope.go:122] "RemoveContainer" containerID="003c79ff01e82d6996fad33e70279321f38e7b857c78e0e44282f87e586bc254"
Oct 31 00:28:25.301492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.300576    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"]
Oct 31 00:28:25.307075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.306076    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7b1fc7-7b8bfdd6f-ssqqm"]
Oct 31 00:28:25.307659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.307628    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:28:25.307820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:25.307677    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:28:26.209641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.209604    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabd2b76-1571-4b37-85bc-5588b382f10f" path="/var/lib/kubelet/pods/cabd2b76-1571-4b37-85bc-5588b382f10f/volumes"
Oct 31 00:28:26.313439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.312992    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:28:26.313439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.313051    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:28:26.615743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.615713    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:26.816249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.816166    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca\") pod \"41ef8c95-01d1-4f94-8f33-efa06bdda094\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") "
Oct 31 00:28:26.816412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.816334    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca\") pod \"41ef8c95-01d1-4f94-8f33-efa06bdda094\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") "
Oct 31 00:28:26.816474 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.816419    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8\" (UniqueName: \"kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8\") pod \"41ef8c95-01d1-4f94-8f33-efa06bdda094\" (UID: \"41ef8c95-01d1-4f94-8f33-efa06bdda094\") "
Oct 31 00:28:26.816600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.816572    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca" pod "41ef8c95-01d1-4f94-8f33-efa06bdda094" (UID: "41ef8c95-01d1-4f94-8f33-efa06bdda094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:26.817121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.817104    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:26.818747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.818720    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca" pod "41ef8c95-01d1-4f94-8f33-efa06bdda094" (UID: "41ef8c95-01d1-4f94-8f33-efa06bdda094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:26.818912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.818887    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8" pod "41ef8c95-01d1-4f94-8f33-efa06bdda094" (UID: "41ef8c95-01d1-4f94-8f33-efa06bdda094"). InnerVolumeSpecName "kube-api-access-kshf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:26.918752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.918713    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ef8c95-01d1-4f94-8f33-efa06bdda094-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:26.918752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:26.918742    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshf8\" (UniqueName: \"kubernetes.io/projected/41ef8c95-01d1-4f94-8f33-efa06bdda094-kube-api-access-kshf8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:27.255203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:27.255177    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"
Oct 31 00:28:27.255378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:27.255211    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg" event={"ID":"41ef8c95-01d1-4f94-8f33-efa06bdda094","Type":"ContainerDied","Data":"3e6431ad71c4533436d93a9ef26906f8ebae2234a07dd61d2d763ed084d3bc19"}
Oct 31 00:28:27.255378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:27.255258    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6431ad71c4533436d93a9ef26906f8ebae2234a07dd61d2d763ed084d3bc19"
Oct 31 00:28:27.308005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:27.307977    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:28:27.308185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:27.308036    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:28:28.307550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:28.307510    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:28:28.307945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:28.307577    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:28:29.308108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.308078    5078 patch_prober.go:29] interesting pod/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 00:28:29.308561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.308143    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 00:28:29.585351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.585260    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"]
Oct 31 00:28:29.586581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.586557    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cabd2b76-1571-4b37-85bc-5588b382f10f" containerName="payload-processing"
Oct 31 00:28:29.586581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.586579    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabd2b76-1571-4b37-85bc-5588b382f10f" containerName="payload-processing"
Oct 31 00:28:29.586805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.586632    5078 cpu_manager.go:414] "containerMap: removing container" podUID="41ef8c95-01d1-4f94-8f33-efa06bdda094" containerName="revoke-keys"
Oct 31 00:28:29.586805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.586640    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ef8c95-01d1-4f94-8f33-efa06bdda094" containerName="revoke-keys"
Oct 31 00:28:29.586912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.586873    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabd2b76-1571-4b37-85bc-5588b382f10f" containerName="payload-processing"
Oct 31 00:28:29.586912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.586891    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ef8c95-01d1-4f94-8f33-efa06bdda094" containerName="revoke-keys"
Oct 31 00:28:29.592164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.592133    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.642624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.642593    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0d4ad9-serving-cert\""
Oct 31 00:28:29.652933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.652908    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"]
Oct 31 00:28:29.701145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.701094    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.701324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.701218    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwtb\" (UniqueName: \"kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.701324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.701313    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.701549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.701526    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.804411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.804377    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.804635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.804613    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.804699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.804667    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwtb\" (UniqueName: \"kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.804763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.804747    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.804966 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:28:29.804950    5078 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-serving-cert: secret "maas-api-e2e-api-b-0d4ad9-serving-cert" not found
Oct 31 00:28:29.805019 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:28:29.805010    5078 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls podName:8c7f7144-264c-4b10-ad57-abbe1bba8fef nodeName:}" failed. No retries permitted until 2026-10-31 00:28:30.304993602 +0000 UTC m=+2078.251364296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls") pod "maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" (UID: "8c7f7144-264c-4b10-ad57-abbe1bba8fef") : secret "maas-api-e2e-api-b-0d4ad9-serving-cert" not found
Oct 31 00:28:29.805210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.805187    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.807860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.807836    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:29.832506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:29.831599    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwtb\" (UniqueName: \"kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:30.312682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.312650    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:28:30.313114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.313051    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:28:30.313389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.313357    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:30.314033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.314001    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:28:30.316288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.316267    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\") pod \"maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:30.556031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.555999    5078 scope.go:122] "RemoveContainer" containerID="f02aeb372dce85181e752422eed1fc994b30e3aa0e0aba76a9b72aaabc09cb5f"
Oct 31 00:28:30.571573 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.571552    5078 scope.go:122] "RemoveContainer" containerID="a884750285dae6d1062db66c4c66288268db1d3cd53e3545ea471190d0ee7b8a"
Oct 31 00:28:30.576098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:30.576069    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:31.038604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:31.038573    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"]
Oct 31 00:28:31.322552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:31.322496    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" event={"ID":"8c7f7144-264c-4b10-ad57-abbe1bba8fef","Type":"ContainerStarted","Data":"52d3c6862fbd7242d81ad9a53634c13ef81d248c98a9e7ca31ad3d74fd233e92"}
Oct 31 00:28:32.352719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:32.352665    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" event={"ID":"8c7f7144-264c-4b10-ad57-abbe1bba8fef","Type":"ContainerStarted","Data":"34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25"}
Oct 31 00:28:32.356892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:32.356860    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:32.409756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:32.409661    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" podStartSLOduration=2.963389597 podStartE2EDuration="3.409642098s" totalImagesPullingTime="446.252501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:32.407748414 +0000 UTC m=+2080.354119156" watchObservedRunningTime="2026-10-31 00:28:32.409642098 +0000 UTC m=+2080.356012818"
Oct 31 00:28:36.392931 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:28:36.392883    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:28:38.384868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:38.384840    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:28:49.331697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:49.331663    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"]
Oct 31 00:28:49.339737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:49.339704    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0df3f8-gb7gg"]
Oct 31 00:28:50.239933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:50.239896    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ef8c95-01d1-4f94-8f33-efa06bdda094" path="/var/lib/kubelet/pods/41ef8c95-01d1-4f94-8f33-efa06bdda094/volumes"
Oct 31 00:28:54.432488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:54.431612    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:28:54.433048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:54.432647    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:28:54.449588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:54.449554    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:28:54.449828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:54.449809    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:28:56.813606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:56.813544    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"]
Oct 31 00:28:56.836128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:56.836091    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7b1fc7-8vp5v"]
Oct 31 00:28:58.237048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:28:58.236987    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30e5dd1-5053-423e-859e-2770c7197f38" path="/var/lib/kubelet/pods/d30e5dd1-5053-423e-859e-2770c7197f38/volumes"
Oct 31 00:29:02.612729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:02.612638    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:29:02.613278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:02.612731    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:04.245838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.245781    5078 generic.go:356] "Generic (PLEG): container finished" podID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerID="444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5" exitCode=0
Oct 31 00:29:04.246528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.245978    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" event={"ID":"603f2b77-a5c8-403a-97e9-c143099f8fb3","Type":"ContainerDied","Data":"444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"}
Oct 31 00:29:04.246630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.246537    5078 scope.go:122] "RemoveContainer" containerID="444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"
Oct 31 00:29:04.248574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.248545    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:29:04.248656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.248616    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:29:04.248656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.248650    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:29:04.705494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.702861    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"]
Oct 31 00:29:04.705494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:04.703100    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" containerID="cri-o://7574b855fe378adf5c0b684ff4677fe8da41102cd5f14c0bd7d10319a59914da" gracePeriod=30
Oct 31 00:29:05.247884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.247849    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:05.248321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.247906    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:05.258912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.258871    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" event={"ID":"603f2b77-a5c8-403a-97e9-c143099f8fb3","Type":"ContainerStarted","Data":"a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7"}
Oct 31 00:29:05.260888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.260862    5078 generic.go:356] "Generic (PLEG): container finished" podID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerID="a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb" exitCode=0
Oct 31 00:29:05.261032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.260939    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" event={"ID":"9f6de8b3-0e20-48c9-8843-cf574e833566","Type":"ContainerDied","Data":"a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"}
Oct 31 00:29:05.261508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.261447    5078 scope.go:122] "RemoveContainer" containerID="a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"
Oct 31 00:29:05.263301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.263269    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:29:05.263417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.263356    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:29:05.263417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.263403    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:29:05.460451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.445148    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"]
Oct 31 00:29:05.460451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:05.447768    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy" containerID="cri-o://0d1c9edcee13b484d7acbe402615452fb74576dbec66b0c1ca4086158eb34e44" gracePeriod=30
Oct 31 00:29:06.209486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.209381    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 00:29:06.209703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.209445    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 00:29:06.218765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.218736    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:29:06.298040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.297996    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" event={"ID":"9f6de8b3-0e20-48c9-8843-cf574e833566","Type":"ContainerStarted","Data":"c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7"}
Oct 31 00:29:06.433161 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:06.432875    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:29:06.518547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.518020    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:29:06.518547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.518168    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:06.518547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:06.518219    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:07.228546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:07.227551    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 00:29:07.228546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:07.227608    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 00:29:07.520541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:07.519996    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:07.520541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:07.520081    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:08.219968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:08.219927    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 00:29:08.221022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:08.220986    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 00:29:08.519168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:08.519075    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:08.519439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:08.519408    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:08.657515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:08.657456    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"]
Oct 31 00:29:08.674493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:08.669645    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"]
Oct 31 00:29:09.212137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:09.212098    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 00:29:09.212930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:09.212888    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 00:29:09.519162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:09.519057    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:09.523758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:09.519446    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:09.972598 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:09.969068    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"]
Oct 31 00:29:10.004779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.004730    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"]
Oct 31 00:29:10.251423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.148615    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 31 00:29:10.364908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.364854    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:10.458675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.404542    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:10.458675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.440591    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 00:29:10.458675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.440660    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 00:29:10.619161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.601179    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 31 00:29:10.619161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.603817    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 31 00:29:10.663604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.658640    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:10.663604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.659203    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:10.663604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.659246    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:10.850160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.849753    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:10.958830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:10.917358    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:11.201063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.149619    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 00:29:11.201063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.149722    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 00:29:11.248492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.212315    5078 patch_prober.go:29] interesting pod/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 00:29:11.248492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.212382    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 00:29:11.324491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.298938    5078 generic.go:356] "Generic (PLEG): container finished" podID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerID="0d1c9edcee13b484d7acbe402615452fb74576dbec66b0c1ca4086158eb34e44" exitCode=0
Oct 31 00:29:11.324491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.299062    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" event={"ID":"9a603b11-c6d9-4c14-932f-c1e3d1454bea","Type":"ContainerDied","Data":"0d1c9edcee13b484d7acbe402615452fb74576dbec66b0c1ca4086158eb34e44"}
Oct 31 00:29:11.324491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.299126    5078 scope.go:122] "RemoveContainer" containerID="4f1c2ec1c5698228c54c5c7e7827d66bed3da07fc9f105c2890f011b3b3a1b4e"
Oct 31 00:29:11.481411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.481280    5078 generic.go:356] "Generic (PLEG): container finished" podID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerID="7574b855fe378adf5c0b684ff4677fe8da41102cd5f14c0bd7d10319a59914da" exitCode=0
Oct 31 00:29:11.481988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.481961    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" event={"ID":"4b20778e-6adf-46f7-ae50-b389aa95f36c","Type":"ContainerDied","Data":"7574b855fe378adf5c0b684ff4677fe8da41102cd5f14c0bd7d10319a59914da"}
Oct 31 00:29:11.517804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.516793    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:11.517804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.516853    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:11.633405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.632322    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:29:11.684488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.672942    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.684488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.672998    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.684488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.673048    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.725491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.724241    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.729471 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.729293    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:11.729706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.729524    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "kube-api-access-gwjm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:11.787295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.777418    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.800820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.800789    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.826488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.801899    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.826866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.826782    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.827065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.827047    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.827778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.827319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket\") pod \"4b20778e-6adf-46f7-ae50-b389aa95f36c\" (UID: \"4b20778e-6adf-46f7-ae50-b389aa95f36c\") "
Oct 31 00:29:11.828529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.828506    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.828661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.828647    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwjm4\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-kube-api-access-gwjm4\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.828758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.828748    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.829160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.795958    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.829276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.803828    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.829361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.804129    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:11.829452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.829135    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.830119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.830090    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.832733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.832674    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token" pod "4b20778e-6adf-46f7-ae50-b389aa95f36c" (UID: "4b20778e-6adf-46f7-ae50-b389aa95f36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:11.852624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.852233    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:29:11.929648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.929617    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.929882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.929850    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.929966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.929913    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.929984    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.930055    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.930119    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.930151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.930286    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.930500    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs\") pod \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\" (UID: \"9a603b11-c6d9-4c14-932f-c1e3d1454bea\") "
Oct 31 00:29:11.930774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.930753    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.931126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.931103    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.931684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.931660    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932320    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b20778e-6adf-46f7-ae50-b389aa95f36c-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932432    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932455    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932492    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b20778e-6adf-46f7-ae50-b389aa95f36c-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932505    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932520    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b20778e-6adf-46f7-ae50-b389aa95f36c-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932534    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932059    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932089    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:11.932568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.932278    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.934116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.934090    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:11.934476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.934420    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:11.934665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.934646    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:11.935869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:11.935840    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86" pod "9a603b11-c6d9-4c14-932f-c1e3d1454bea" (UID: "9a603b11-c6d9-4c14-932f-c1e3d1454bea"). InnerVolumeSpecName "kube-api-access-cnd86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:12.034114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034016    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.034114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034059    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnd86\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-kube-api-access-cnd86\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.034114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034074    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.034114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034086    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.034114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034099    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.034114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034111    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.034447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.034123    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a603b11-c6d9-4c14-932f-c1e3d1454bea-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.225760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.225725    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:29:12.226282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.226260    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:29:12.227022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.227003    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:29:12.529544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.529500    5078 patch_prober.go:29] interesting pod/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 00:29:12.530123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.530094    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 00:29:12.550325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.550294    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2" event={"ID":"9a603b11-c6d9-4c14-932f-c1e3d1454bea","Type":"ContainerDied","Data":"e89be8bbe624deb5853e41532befdd5e7ba5aaeaf5f0b2ec818655f4dc92077f"}
Oct 31 00:29:12.550579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.550567    5078 scope.go:122] "RemoveContainer" containerID="0d1c9edcee13b484d7acbe402615452fb74576dbec66b0c1ca4086158eb34e44"
Oct 31 00:29:12.550676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.550658    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"
Oct 31 00:29:12.554119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.554099    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z" event={"ID":"4b20778e-6adf-46f7-ae50-b389aa95f36c","Type":"ContainerDied","Data":"bf4fa4bced80e756bbdf2237523a96f869d2b7ba9278bb4c35a829dc9ec2372d"}
Oct 31 00:29:12.554502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.554489    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"
Oct 31 00:29:12.610578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.610508    5078 scope.go:122] "RemoveContainer" containerID="7574b855fe378adf5c0b684ff4677fe8da41102cd5f14c0bd7d10319a59914da"
Oct 31 00:29:12.631060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.630965    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"]
Oct 31 00:29:12.638576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.638554    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7b1fc7-openshift-default-6d7bd6dc95-qrdl2"]
Oct 31 00:29:12.655848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.655826    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"]
Oct 31 00:29:12.662516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:12.662499    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-ln89z"]
Oct 31 00:29:13.662406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.662369    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing" containerID="cri-o://bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed" gracePeriod=25
Oct 31 00:29:13.686455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.686417    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing" containerID="cri-o://de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8" gracePeriod=25
Oct 31 00:29:13.687570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.687550    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:29:13.702019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.701689    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:29:13.711580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.710599    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:29:13.808358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.808328    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"]
Oct 31 00:29:13.851411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:13.851049    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d609e2-lc6sp"]
Oct 31 00:29:14.154711 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.154677    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:29:14.204762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.204689    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj\" (UniqueName: \"kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj\") pod \"a75e181d-6850-4b90-a34a-3a1d098a6f53\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") "
Oct 31 00:29:14.204917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.204772    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume\") pod \"a75e181d-6850-4b90-a34a-3a1d098a6f53\" (UID: \"a75e181d-6850-4b90-a34a-3a1d098a6f53\") "
Oct 31 00:29:14.206379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.206350    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume" pod "a75e181d-6850-4b90-a34a-3a1d098a6f53" (UID: "a75e181d-6850-4b90-a34a-3a1d098a6f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:14.214717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.214690    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj" pod "a75e181d-6850-4b90-a34a-3a1d098a6f53" (UID: "a75e181d-6850-4b90-a34a-3a1d098a6f53"). InnerVolumeSpecName "kube-api-access-rhvrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:14.216683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.216654    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:29:14.218338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.218308    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" path="/var/lib/kubelet/pods/4b20778e-6adf-46f7-ae50-b389aa95f36c/volumes"
Oct 31 00:29:14.219090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.219070    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65495e5a-7cfc-4f3a-841c-c4df05c2a8d7" path="/var/lib/kubelet/pods/65495e5a-7cfc-4f3a-841c-c4df05c2a8d7/volumes"
Oct 31 00:29:14.219820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.219794    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" path="/var/lib/kubelet/pods/9a603b11-c6d9-4c14-932f-c1e3d1454bea/volumes"
Oct 31 00:29:14.311387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.311339    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2\" (UniqueName: \"kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2\") pod \"325f8eed-03c3-4f20-a72a-7753e560cb45\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") "
Oct 31 00:29:14.311646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.311409    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume\") pod \"325f8eed-03c3-4f20-a72a-7753e560cb45\" (UID: \"325f8eed-03c3-4f20-a72a-7753e560cb45\") "
Oct 31 00:29:14.312941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.312713    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvrj\" (UniqueName: \"kubernetes.io/projected/a75e181d-6850-4b90-a34a-3a1d098a6f53-kube-api-access-rhvrj\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:14.313177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.313140    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume" pod "325f8eed-03c3-4f20-a72a-7753e560cb45" (UID: "325f8eed-03c3-4f20-a72a-7753e560cb45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:14.313271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.313198    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75e181d-6850-4b90-a34a-3a1d098a6f53-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:14.320444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.317612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2" pod "325f8eed-03c3-4f20-a72a-7753e560cb45" (UID: "325f8eed-03c3-4f20-a72a-7753e560cb45"). InnerVolumeSpecName "kube-api-access-wnwq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:14.415521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.415448    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnwq2\" (UniqueName: \"kubernetes.io/projected/325f8eed-03c3-4f20-a72a-7753e560cb45-kube-api-access-wnwq2\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:14.415521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.415502    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325f8eed-03c3-4f20-a72a-7753e560cb45-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:14.643975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.643944    5078 generic.go:356] "Generic (PLEG): container finished" podID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerID="bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed" exitCode=0
Oct 31 00:29:14.644147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.644011    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"
Oct 31 00:29:14.644147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.644042    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" event={"ID":"a75e181d-6850-4b90-a34a-3a1d098a6f53","Type":"ContainerDied","Data":"bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed"}
Oct 31 00:29:14.644147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.644102    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6" event={"ID":"a75e181d-6850-4b90-a34a-3a1d098a6f53","Type":"ContainerDied","Data":"1aa6b03a0a507a5333aadfeac6d0c12911a98e642e830b0781bf7e93691322b8"}
Oct 31 00:29:14.644147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.644130    5078 scope.go:122] "RemoveContainer" containerID="bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed"
Oct 31 00:29:14.646015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.645988    5078 generic.go:356] "Generic (PLEG): container finished" podID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerID="de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8" exitCode=0
Oct 31 00:29:14.646167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.646029    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"
Oct 31 00:29:14.646167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.646070    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" event={"ID":"325f8eed-03c3-4f20-a72a-7753e560cb45","Type":"ContainerDied","Data":"de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8"}
Oct 31 00:29:14.646167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.646113    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p" event={"ID":"325f8eed-03c3-4f20-a72a-7753e560cb45","Type":"ContainerDied","Data":"bd838dce2f58387c9e899064122e377a501788cf0a5bfeaded1bdd2d03f49b3d"}
Oct 31 00:29:14.659092 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.659067    5078 scope.go:122] "RemoveContainer" containerID="bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed"
Oct 31 00:29:14.659530 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:14.659438    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed\": container with ID starting with bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed not found: ID does not exist" containerID="bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed"
Oct 31 00:29:14.659677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.659650    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed"} err="failed to get container status \"bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed\": rpc error: code = NotFound desc = could not find container \"bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed\": container with ID starting with bb62f781d71c1f060c6785a6110bc15103144d5350cd1cd2070ec281abfd36ed not found: ID does not exist"
Oct 31 00:29:14.659677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.659676    5078 scope.go:122] "RemoveContainer" containerID="de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8"
Oct 31 00:29:14.671852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.671836    5078 scope.go:122] "RemoveContainer" containerID="de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8"
Oct 31 00:29:14.672168 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:14.672147    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8\": container with ID starting with de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8 not found: ID does not exist" containerID="de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8"
Oct 31 00:29:14.672213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.672175    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8"} err="failed to get container status \"de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8\": rpc error: code = NotFound desc = could not find container \"de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8\": container with ID starting with de47468352669f6b755cbc10744b4189da709e1bd8a6c6ad2fe42af5ca6fa0b8 not found: ID does not exist"
Oct 31 00:29:14.685246 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.685223    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"]
Oct 31 00:29:14.698172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.698150    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a7688-866955dc6c-8pt4p"]
Oct 31 00:29:14.725765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.725737    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"]
Oct 31 00:29:14.731334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:14.731305    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a7688-7db694fc99-k9hn6"]
Oct 31 00:29:15.005811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.005718    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4" podUID="c0429edc-ac06-4abe-8811-31047ba93cf4" containerName="payload-processing" containerID="cri-o://20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424" gracePeriod=25
Oct 31 00:29:15.005811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.005735    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb" podUID="55fa4ee3-9bfd-43ac-97c6-9448a4635262" containerName="payload-pre-processing" containerID="cri-o://3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451" gracePeriod=25
Oct 31 00:29:15.412713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.412691    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:29:15.422007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.421976    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:29:15.534412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.534328    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume\") pod \"c0429edc-ac06-4abe-8811-31047ba93cf4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") "
Oct 31 00:29:15.534412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.534388    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv\" (UniqueName: \"kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv\") pod \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") "
Oct 31 00:29:15.534686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.534433    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm\" (UniqueName: \"kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm\") pod \"c0429edc-ac06-4abe-8811-31047ba93cf4\" (UID: \"c0429edc-ac06-4abe-8811-31047ba93cf4\") "
Oct 31 00:29:15.534755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.534728    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume" pod "c0429edc-ac06-4abe-8811-31047ba93cf4" (UID: "c0429edc-ac06-4abe-8811-31047ba93cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:15.534818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.534750    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume\") pod \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\" (UID: \"55fa4ee3-9bfd-43ac-97c6-9448a4635262\") "
Oct 31 00:29:15.535073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.535052    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume" pod "55fa4ee3-9bfd-43ac-97c6-9448a4635262" (UID: "55fa4ee3-9bfd-43ac-97c6-9448a4635262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:15.535909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.535881    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fa4ee3-9bfd-43ac-97c6-9448a4635262-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:15.536021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.535912    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0429edc-ac06-4abe-8811-31047ba93cf4-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:15.536863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.536840    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm" pod "c0429edc-ac06-4abe-8811-31047ba93cf4" (UID: "c0429edc-ac06-4abe-8811-31047ba93cf4"). InnerVolumeSpecName "kube-api-access-vspfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:15.537022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.536999    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv" pod "55fa4ee3-9bfd-43ac-97c6-9448a4635262" (UID: "55fa4ee3-9bfd-43ac-97c6-9448a4635262"). InnerVolumeSpecName "kube-api-access-m2cmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:15.637849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.637819    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2cmv\" (UniqueName: \"kubernetes.io/projected/55fa4ee3-9bfd-43ac-97c6-9448a4635262-kube-api-access-m2cmv\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:15.637849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.637847    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vspfm\" (UniqueName: \"kubernetes.io/projected/c0429edc-ac06-4abe-8811-31047ba93cf4-kube-api-access-vspfm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:15.657556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.657520    5078 generic.go:356] "Generic (PLEG): container finished" podID="c0429edc-ac06-4abe-8811-31047ba93cf4" containerID="20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424" exitCode=0
Oct 31 00:29:15.657725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.657580    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"
Oct 31 00:29:15.657725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.657635    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4" event={"ID":"c0429edc-ac06-4abe-8811-31047ba93cf4","Type":"ContainerDied","Data":"20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424"}
Oct 31 00:29:15.657725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.657695    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4" event={"ID":"c0429edc-ac06-4abe-8811-31047ba93cf4","Type":"ContainerDied","Data":"dc7989c4eff0862a338bc466c2985b4635a1fa393006d745a329022d33cadda6"}
Oct 31 00:29:15.657725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.657723    5078 scope.go:122] "RemoveContainer" containerID="20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424"
Oct 31 00:29:15.660422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.660399    5078 generic.go:356] "Generic (PLEG): container finished" podID="55fa4ee3-9bfd-43ac-97c6-9448a4635262" containerID="3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451" exitCode=0
Oct 31 00:29:15.660712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.660449    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"
Oct 31 00:29:15.660836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.660498    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb" event={"ID":"55fa4ee3-9bfd-43ac-97c6-9448a4635262","Type":"ContainerDied","Data":"3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451"}
Oct 31 00:29:15.660836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.660807    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb" event={"ID":"55fa4ee3-9bfd-43ac-97c6-9448a4635262","Type":"ContainerDied","Data":"f9d30cac3a60ccfeff6394bd44a8242def1262095dd3fef8f416df2dea96f632"}
Oct 31 00:29:15.676624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.676608    5078 scope.go:122] "RemoveContainer" containerID="20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424"
Oct 31 00:29:15.677570 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:15.677543    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424\": container with ID starting with 20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424 not found: ID does not exist" containerID="20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424"
Oct 31 00:29:15.677696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.677578    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424"} err="failed to get container status \"20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424\": rpc error: code = NotFound desc = could not find container \"20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424\": container with ID starting with 20a077a4bbb0d50947b3b687f11f774a43db386856e26fa4a28f4d5f0e03a424 not found: ID does not exist"
Oct 31 00:29:15.677696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.677603    5078 scope.go:122] "RemoveContainer" containerID="3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451"
Oct 31 00:29:15.691612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.691590    5078 scope.go:122] "RemoveContainer" containerID="3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451"
Oct 31 00:29:15.692020 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:15.691994    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451\": container with ID starting with 3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451 not found: ID does not exist" containerID="3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451"
Oct 31 00:29:15.692096 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.692042    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451"} err="failed to get container status \"3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451\": rpc error: code = NotFound desc = could not find container \"3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451\": container with ID starting with 3ffdd1b4bae56aaec51bed1edc6cd93e81088e05bf6f9f2bc550a12207aac451 not found: ID does not exist"
Oct 31 00:29:15.694808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.694787    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"]
Oct 31 00:29:15.700579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.700558    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc0fe1-7d648c879-2z9fb"]
Oct 31 00:29:15.718510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.718447    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"]
Oct 31 00:29:15.723211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:15.723189    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc0fe1-758497cf59-6lcp4"]
Oct 31 00:29:16.299115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:16.299076    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" path="/var/lib/kubelet/pods/325f8eed-03c3-4f20-a72a-7753e560cb45/volumes"
Oct 31 00:29:16.299833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:16.299804    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fa4ee3-9bfd-43ac-97c6-9448a4635262" path="/var/lib/kubelet/pods/55fa4ee3-9bfd-43ac-97c6-9448a4635262/volumes"
Oct 31 00:29:16.300592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:16.300561    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" path="/var/lib/kubelet/pods/a75e181d-6850-4b90-a34a-3a1d098a6f53/volumes"
Oct 31 00:29:16.301200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:16.301181    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0429edc-ac06-4abe-8811-31047ba93cf4" path="/var/lib/kubelet/pods/c0429edc-ac06-4abe-8811-31047ba93cf4/volumes"
Oct 31 00:29:20.827295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:20.827254    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"]
Oct 31 00:29:20.827823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:20.827527    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api" containerID="cri-o://5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b" gracePeriod=30
Oct 31 00:29:21.440773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.440749    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:29:21.561764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.561733    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs\") pod \"010f4cfc-84d4-4b18-a9ff-289488077e9b\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") "
Oct 31 00:29:21.561932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.561783    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca\") pod \"010f4cfc-84d4-4b18-a9ff-289488077e9b\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") "
Oct 31 00:29:21.561932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.561859    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr\" (UniqueName: \"kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr\") pod \"010f4cfc-84d4-4b18-a9ff-289488077e9b\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") "
Oct 31 00:29:21.562264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.562242    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls\") pod \"010f4cfc-84d4-4b18-a9ff-289488077e9b\" (UID: \"010f4cfc-84d4-4b18-a9ff-289488077e9b\") "
Oct 31 00:29:21.563806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.563777    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca" pod "010f4cfc-84d4-4b18-a9ff-289488077e9b" (UID: "010f4cfc-84d4-4b18-a9ff-289488077e9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:21.567349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.567306    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr" pod "010f4cfc-84d4-4b18-a9ff-289488077e9b" (UID: "010f4cfc-84d4-4b18-a9ff-289488077e9b"). InnerVolumeSpecName "kube-api-access-lqxhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:21.593116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.593068    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs" pod "010f4cfc-84d4-4b18-a9ff-289488077e9b" (UID: "010f4cfc-84d4-4b18-a9ff-289488077e9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:21.593617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.593560    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls" pod "010f4cfc-84d4-4b18-a9ff-289488077e9b" (UID: "010f4cfc-84d4-4b18-a9ff-289488077e9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:21.664949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.664874    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:21.664949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.664904    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010f4cfc-84d4-4b18-a9ff-289488077e9b-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:21.664949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.664918    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxhr\" (UniqueName: \"kubernetes.io/projected/010f4cfc-84d4-4b18-a9ff-289488077e9b-kube-api-access-lqxhr\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:21.664949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.664930    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010f4cfc-84d4-4b18-a9ff-289488077e9b-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:21.814487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.813817    5078 generic.go:356] "Generic (PLEG): container finished" podID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerID="5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b" exitCode=0
Oct 31 00:29:21.814487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.814018    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" event={"ID":"010f4cfc-84d4-4b18-a9ff-289488077e9b","Type":"ContainerDied","Data":"5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b"}
Oct 31 00:29:21.814487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.814083    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r" event={"ID":"010f4cfc-84d4-4b18-a9ff-289488077e9b","Type":"ContainerDied","Data":"dcabb5fa933d7729e8719bf6622bfd6ed5b4feaca6a94a5f88cd084362a52097"}
Oct 31 00:29:21.814487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.814112    5078 scope.go:122] "RemoveContainer" containerID="5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b"
Oct 31 00:29:21.814487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.814288    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"
Oct 31 00:29:21.976348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.976319    5078 scope.go:122] "RemoveContainer" containerID="5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b"
Oct 31 00:29:21.976853 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:21.976808    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b\": container with ID starting with 5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b not found: ID does not exist" containerID="5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b"
Oct 31 00:29:21.976924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:21.976844    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b"} err="failed to get container status \"5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b\": rpc error: code = NotFound desc = could not find container \"5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b\": container with ID starting with 5c6d7f1e12e35a51bbe7ac3b0fe4d8f28e36b1e67bc4ca2aa56758a9ed51551b not found: ID does not exist"
Oct 31 00:29:22.014817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:22.014784    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"]
Oct 31 00:29:22.019415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:22.019376    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-41a0b7-9db9d77b5-vld9r"]
Oct 31 00:29:22.199909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:22.199874    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" path="/var/lib/kubelet/pods/010f4cfc-84d4-4b18-a9ff-289488077e9b/volumes"
Oct 31 00:29:26.268168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:26.268101    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"]
Oct 31 00:29:26.268751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:26.268456    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy" containerID="cri-o://9f126349b71a0ed63e287d00793de4ff76ead0d073f93537763560b26d45af8b" gracePeriod=30
Oct 31 00:29:26.283099 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:26.283065    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"]
Oct 31 00:29:26.283319 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:26.283290    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy" containerID="cri-o://e3984bf2c70f13c453fed8693fd631f9e17d6ed6f2bf4275e7439ea20ce9466d" gracePeriod=30
Oct 31 00:29:26.287354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:26.287322    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"]
Oct 31 00:29:26.287657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:26.287627    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy" containerID="cri-o://f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1" gracePeriod=30
Oct 31 00:29:29.265815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:29.265778    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"]
Oct 31 00:29:29.266325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:29.266031    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy" containerID="cri-o://c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a" gracePeriod=30
Oct 31 00:29:31.824574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.824244    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"]
Oct 31 00:29:31.839924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.839880    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"]
Oct 31 00:29:31.849834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.849761    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:29:31.918053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.918013    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.918304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.918286    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.918562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.918545    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.918708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.918693    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.918855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.918841    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.919130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.919112    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.919628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.919604    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.919799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.919785    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.921529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.921508    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff\") pod \"0038939b-91e8-4b4c-848f-2da60082ff8e\" (UID: \"0038939b-91e8-4b4c-848f-2da60082ff8e\") "
Oct 31 00:29:31.933865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.933830    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:31.940868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.937380    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:31.941631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:31.941569    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.002014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.001972    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.002278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.002250    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.002387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.002341    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.002447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.002393    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025743    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025773    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025789    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025802    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025816    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025830    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0038939b-91e8-4b4c-848f-2da60082ff8e-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.026994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.025844    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.086068 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.085995    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "kube-api-access-vnmff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.197521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.197424    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnmff\" (UniqueName: \"kubernetes.io/projected/0038939b-91e8-4b4c-848f-2da60082ff8e-kube-api-access-vnmff\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.260863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.260831    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:29:32.280888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.280859    5078 generic.go:356] "Generic (PLEG): container finished" podID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerID="9f126349b71a0ed63e287d00793de4ff76ead0d073f93537763560b26d45af8b" exitCode=0
Oct 31 00:29:32.281256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.281233    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" event={"ID":"86417f75-420a-4ef6-84ee-1d36bd8632ca","Type":"ContainerDied","Data":"9f126349b71a0ed63e287d00793de4ff76ead0d073f93537763560b26d45af8b"}
Oct 31 00:29:32.281454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.281408    5078 scope.go:122] "RemoveContainer" containerID="5a155a3b85f1461cac00c73db5137ff20b569a162b1005b5426884a75e779b2f"
Oct 31 00:29:32.285489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.285444    5078 generic.go:356] "Generic (PLEG): container finished" podID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerID="e3984bf2c70f13c453fed8693fd631f9e17d6ed6f2bf4275e7439ea20ce9466d" exitCode=0
Oct 31 00:29:32.286000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.285983    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:29:32.286315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.286300    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"
Oct 31 00:29:32.286676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.286652    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" event={"ID":"5a23b543-84bf-4a68-9e80-414b3fdfc4ea","Type":"ContainerDied","Data":"e3984bf2c70f13c453fed8693fd631f9e17d6ed6f2bf4275e7439ea20ce9466d"}
Oct 31 00:29:32.289137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.289107    5078 generic.go:356] "Generic (PLEG): container finished" podID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerID="f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1" exitCode=0
Oct 31 00:29:32.289455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.289436    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" event={"ID":"0038939b-91e8-4b4c-848f-2da60082ff8e","Type":"ContainerDied","Data":"f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1"}
Oct 31 00:29:32.289623 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.289605    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr" event={"ID":"0038939b-91e8-4b4c-848f-2da60082ff8e","Type":"ContainerDied","Data":"e943288e5db6ff3cf1acbe130c0c3e021a72ffea51b64c875577449fb9c58a3f"}
Oct 31 00:29:32.289907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.289890    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"
Oct 31 00:29:32.527546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.470649    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.539605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.539566    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.547484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.539915    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.567518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.547675    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.568852 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.568825    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.569044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.569027    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.569167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.569154    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.569273 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.569259    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.569387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.569374    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.652139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.564169    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.652139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.593690    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:32.652139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.643610    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.652139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.644810    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.718221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.714666    5078 scope.go:122] "RemoveContainer" containerID="e3984bf2c70f13c453fed8693fd631f9e17d6ed6f2bf4275e7439ea20ce9466d"
Oct 31 00:29:32.724079 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.724056    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.765633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.765608    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.768113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.768060    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.768634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.750571    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.768634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.751610    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo" pod "0038939b-91e8-4b4c-848f-2da60082ff8e" (UID: "0038939b-91e8-4b4c-848f-2da60082ff8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:32.768634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.756450    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.769102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.769082    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.769688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.769575    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:32.781517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.776414    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.781713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.781694    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.796216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.796196    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs\") pod \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\" (UID: \"5a23b543-84bf-4a68-9e80-414b3fdfc4ea\") "
Oct 31 00:29:32.829786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.796421    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.833453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.833428    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs\") pod \"86417f75-420a-4ef6-84ee-1d36bd8632ca\" (UID: \"86417f75-420a-4ef6-84ee-1d36bd8632ca\") "
Oct 31 00:29:32.848481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.812270    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.848481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.833208    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.862930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.862294    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.862930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.862366    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.900629    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "kube-api-access-mkb5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.902990    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.906239    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.869035    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907041    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86417f75-420a-4ef6-84ee-1d36bd8632ca-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907065    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907080    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907115    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0038939b-91e8-4b4c-848f-2da60082ff8e-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907132    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907147    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907163    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907196    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.907212    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.914197    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "kube-api-access-fcfnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.918688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.914275    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.965486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:32.927222    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo" pod "5a23b543-84bf-4a68-9e80-414b3fdfc4ea" (UID: "5a23b543-84bf-4a68-9e80-414b3fdfc4ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.009968    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010007    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010026    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkb5b\" (UniqueName: \"kubernetes.io/projected/86417f75-420a-4ef6-84ee-1d36bd8632ca-kube-api-access-mkb5b\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010051    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcfnp\" (UniqueName: \"kubernetes.io/projected/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-kube-api-access-fcfnp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010067    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010081    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010095    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.010109    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a23b543-84bf-4a68-9e80-414b3fdfc4ea-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.090830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.022993    5078 scope.go:122] "RemoveContainer" containerID="f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1"
Oct 31 00:29:33.250146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.250107    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy" pod "86417f75-420a-4ef6-84ee-1d36bd8632ca" (UID: "86417f75-420a-4ef6-84ee-1d36bd8632ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:33.529531 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.331071    5078 scope.go:122] "RemoveContainer" containerID="f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1"
Oct 31 00:29:33.529862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.331211    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.548053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.470604    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs" podUID="7aef86cd-1369-4407-a6a7-4febffe6d133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.548053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.499606    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.548331 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:33.548138    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1\": container with ID starting with f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1 not found: ID does not exist" containerID="f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1"
Oct 31 00:29:33.548331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.548180    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1"} err="failed to get container status \"f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1\": rpc error: code = NotFound desc = could not find container \"f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1\": container with ID starting with f6bfda33f7f314325350e8c60d78edabef8b49922ddb954e2fb299c8c94ebda1 not found: ID does not exist"
Oct 31 00:29:33.548956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.548911    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"]
Oct 31 00:29:33.639518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.619057    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.639518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.621376    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86417f75-420a-4ef6-84ee-1d36bd8632ca-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.689257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.652912    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.717640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.717601    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.761601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.761560    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.846486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.845946    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.846486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.846012    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.882707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.863891    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded"
Oct 31 00:29:33.882707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.871836    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"
Oct 31 00:29:33.882707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.872810    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-64bnr"]
Oct 31 00:29:33.882707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.872854    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh" event={"ID":"86417f75-420a-4ef6-84ee-1d36bd8632ca","Type":"ContainerDied","Data":"6a1654258341307c9004b3aa76426c1c2c80c0e7370ea75b683132f8466e4c75"}
Oct 31 00:29:33.882707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.872905    5078 scope.go:122] "RemoveContainer" containerID="9f126349b71a0ed63e287d00793de4ff76ead0d073f93537763560b26d45af8b"
Oct 31 00:29:33.908555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.900698    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"]
Oct 31 00:29:33.908555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.901154    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:29:33.908555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:33.903404    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4" event={"ID":"5a23b543-84bf-4a68-9e80-414b3fdfc4ea","Type":"ContainerDied","Data":"dd6761d98889d5adef8c8f2ef6911ab65453f3d5de0e4995c93c49889b2d9554"}
Oct 31 00:29:34.059489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.058514    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-fb4x4"]
Oct 31 00:29:34.125893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.125501    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"]
Oct 31 00:29:34.164486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.155093    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-kgwzh"]
Oct 31 00:29:34.211578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.209500    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" path="/var/lib/kubelet/pods/0038939b-91e8-4b4c-848f-2da60082ff8e/volumes"
Oct 31 00:29:34.211578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.210266    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" path="/var/lib/kubelet/pods/5a23b543-84bf-4a68-9e80-414b3fdfc4ea/volumes"
Oct 31 00:29:34.211578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.210932    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" path="/var/lib/kubelet/pods/86417f75-420a-4ef6-84ee-1d36bd8632ca/volumes"
Oct 31 00:29:34.668928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.668899    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:29:34.727830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.727784    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.727854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.727901    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.727958    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.728200    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.728251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.728385    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.728503    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.728645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.728572    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy\") pod \"86073226-02be-4443-b886-654281e0a89a\" (UID: \"86073226-02be-4443-b886-654281e0a89a\") "
Oct 31 00:29:34.729747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.729701    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:34.729957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.729922    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.730225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.730197    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:34.730833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.730807    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:34.731225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.731207    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:34.762782 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.758818    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:34.764045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.764004    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:34.788744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.788712    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:34.789007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.788988    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "kube-api-access-9gnqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:34.790362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.790336    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy" pod "86073226-02be-4443-b886-654281e0a89a" (UID: "86073226-02be-4443-b886-654281e0a89a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:34.839274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839235    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839277    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839293    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839307    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839322    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86073226-02be-4443-b886-654281e0a89a-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839337    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/86073226-02be-4443-b886-654281e0a89a-kube-api-access-9gnqp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839353    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86073226-02be-4443-b886-654281e0a89a-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.839551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.839367    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86073226-02be-4443-b886-654281e0a89a-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:34.929656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.929481    5078 generic.go:356] "Generic (PLEG): container finished" podID="86073226-02be-4443-b886-654281e0a89a" containerID="c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a" exitCode=0
Oct 31 00:29:34.929656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.929560    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"
Oct 31 00:29:34.929656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.929593    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" event={"ID":"86073226-02be-4443-b886-654281e0a89a","Type":"ContainerDied","Data":"c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a"}
Oct 31 00:29:34.930303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.930080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk" event={"ID":"86073226-02be-4443-b886-654281e0a89a","Type":"ContainerDied","Data":"7525b31ee764f02ad69c0c74f01675dbaff45b5380a97db4e75ab02565779b51"}
Oct 31 00:29:34.930303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.930127    5078 scope.go:122] "RemoveContainer" containerID="c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a"
Oct 31 00:29:34.945372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.945351    5078 scope.go:122] "RemoveContainer" containerID="e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"
Oct 31 00:29:34.956608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.956586    5078 scope.go:122] "RemoveContainer" containerID="c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a"
Oct 31 00:29:34.956886 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:34.956864    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a\": container with ID starting with c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a not found: ID does not exist" containerID="c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a"
Oct 31 00:29:34.956972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.956897    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a"} err="failed to get container status \"c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a\": rpc error: code = NotFound desc = could not find container \"c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a\": container with ID starting with c560f67291245d2e7c0019cc528e91f7e233e60a71e01d700fe9fdf86daa441a not found: ID does not exist"
Oct 31 00:29:34.956972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.956934    5078 scope.go:122] "RemoveContainer" containerID="e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"
Oct 31 00:29:34.957177 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:34.957159    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec\": container with ID starting with e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec not found: ID does not exist" containerID="e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"
Oct 31 00:29:34.957230 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.957186    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec"} err="failed to get container status \"e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec\": rpc error: code = NotFound desc = could not find container \"e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec\": container with ID starting with e2c39c47485d320cab8dafcfdf0d201e5f8766191dc9564fa44947d5e4e96cec not found: ID does not exist"
Oct 31 00:29:34.987982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.987934    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"]
Oct 31 00:29:34.992685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:34.992654    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-41a0b7-openshift-default-5cf7fc4dfc-nkxbk"]
Oct 31 00:29:36.203225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:36.203180    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86073226-02be-4443-b886-654281e0a89a" path="/var/lib/kubelet/pods/86073226-02be-4443-b886-654281e0a89a/volumes"
Oct 31 00:29:36.396478 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:36.396426    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:29:40.025714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:40.025680    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"]
Oct 31 00:29:42.542662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:42.542630    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"]
Oct 31 00:29:42.561974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:42.561928    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"]
Oct 31 00:29:45.026241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.026196    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing" containerID="cri-o://0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b" gracePeriod=25
Oct 31 00:29:45.366412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.366386    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:29:45.373061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.373029    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2\" (UniqueName: \"kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2\") pod \"cc4de92e-ac35-496d-9c11-297373884443\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") "
Oct 31 00:29:45.373293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.373271    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume\") pod \"cc4de92e-ac35-496d-9c11-297373884443\" (UID: \"cc4de92e-ac35-496d-9c11-297373884443\") "
Oct 31 00:29:45.373957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.373928    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume" pod "cc4de92e-ac35-496d-9c11-297373884443" (UID: "cc4de92e-ac35-496d-9c11-297373884443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:45.376119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.376095    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2" pod "cc4de92e-ac35-496d-9c11-297373884443" (UID: "cc4de92e-ac35-496d-9c11-297373884443"). InnerVolumeSpecName "kube-api-access-kp5m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:45.475565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.475524    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5m2\" (UniqueName: \"kubernetes.io/projected/cc4de92e-ac35-496d-9c11-297373884443-kube-api-access-kp5m2\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:45.475565 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:45.475561    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de92e-ac35-496d-9c11-297373884443-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:46.120713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.120673    5078 generic.go:356] "Generic (PLEG): container finished" podID="cc4de92e-ac35-496d-9c11-297373884443" containerID="0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b" exitCode=0
Oct 31 00:29:46.121175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.120742    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"
Oct 31 00:29:46.121175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.120741    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" event={"ID":"cc4de92e-ac35-496d-9c11-297373884443","Type":"ContainerDied","Data":"0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b"}
Oct 31 00:29:46.121175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.120866    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv" event={"ID":"cc4de92e-ac35-496d-9c11-297373884443","Type":"ContainerDied","Data":"844e0ace63306c759dc814b421441bdc35991a53f2b4a85b72ae820ce2a38836"}
Oct 31 00:29:46.121175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.120895    5078 scope.go:122] "RemoveContainer" containerID="0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b"
Oct 31 00:29:46.138814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.138347    5078 scope.go:122] "RemoveContainer" containerID="0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b"
Oct 31 00:29:46.138814 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:46.138703    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b\": container with ID starting with 0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b not found: ID does not exist" containerID="0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b"
Oct 31 00:29:46.138814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.138730    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b"} err="failed to get container status \"0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b\": rpc error: code = NotFound desc = could not find container \"0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b\": container with ID starting with 0d346d4e05d9e5e956a6fd5987fc3ebf6493954e92b594c6dbeb13535968807b not found: ID does not exist"
Oct 31 00:29:46.151891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.151869    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"]
Oct 31 00:29:46.156154 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.156130    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a39be4-775b5d5b5f-4bzvv"]
Oct 31 00:29:46.198787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:46.198759    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4de92e-ac35-496d-9c11-297373884443" path="/var/lib/kubelet/pods/cc4de92e-ac35-496d-9c11-297373884443/volumes"
Oct 31 00:29:47.076869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.076827    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main" containerID="cri-o://e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7" gracePeriod=45
Oct 31 00:29:47.474452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.474419    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:29:47.510734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.510700    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm\") pod \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") "
Oct 31 00:29:47.510934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.510822    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh\" (UniqueName: \"kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh\") pod \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") "
Oct 31 00:29:47.511031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.511014    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home\") pod \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") "
Oct 31 00:29:47.511150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.511132    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache\") pod \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") "
Oct 31 00:29:47.511257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.511240    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs\" (UniqueName: \"kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs\") pod \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") "
Oct 31 00:29:47.511394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.511309    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir\") pod \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\" (UID: \"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f\") "
Oct 31 00:29:47.512398 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.512366    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir" pod "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" (UID: "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:47.519540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.515354    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm" pod "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" (UID: "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:47.521296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.520690    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh" pod "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" (UID: "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f"). InnerVolumeSpecName "kube-api-access-x82fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:47.521296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.520913    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home" pod "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" (UID: "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:47.521296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.521225    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache" pod "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" (UID: "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:47.523704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.523657    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs" pod "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" (UID: "75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:47.543677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.543609    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q" podUID="bc629a17-3d53-4c71-8ca9-8d140ee3618a" containerName="payload-processing" containerID="cri-o://a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0" gracePeriod=25
Oct 31 00:29:47.562284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.562248    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r" podUID="0165dc8b-2e01-4dba-a5bc-04c847535b64" containerName="payload-pre-processing" containerID="cri-o://da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818" gracePeriod=25
Oct 31 00:29:47.614144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.614115    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.614144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.614144    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.614331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.614160    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.614331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.614172    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.614331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.614184    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.614331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.614196    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x82fh\" (UniqueName: \"kubernetes.io/projected/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f-kube-api-access-x82fh\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.898203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.898172    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:29:47.921513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.921435    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume\") pod \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") "
Oct 31 00:29:47.921689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.921603    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9\" (UniqueName: \"kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9\") pod \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\" (UID: \"bc629a17-3d53-4c71-8ca9-8d140ee3618a\") "
Oct 31 00:29:47.922015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.921987    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume" pod "bc629a17-3d53-4c71-8ca9-8d140ee3618a" (UID: "bc629a17-3d53-4c71-8ca9-8d140ee3618a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:47.922689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.922654    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc629a17-3d53-4c71-8ca9-8d140ee3618a-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:47.924763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.924740    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9" pod "bc629a17-3d53-4c71-8ca9-8d140ee3618a" (UID: "bc629a17-3d53-4c71-8ca9-8d140ee3618a"). InnerVolumeSpecName "kube-api-access-zkxf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:47.995631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:47.995563    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:29:48.029096 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.029052    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume\") pod \"0165dc8b-2e01-4dba-a5bc-04c847535b64\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") "
Oct 31 00:29:48.029392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.029369    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5\" (UniqueName: \"kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5\") pod \"0165dc8b-2e01-4dba-a5bc-04c847535b64\" (UID: \"0165dc8b-2e01-4dba-a5bc-04c847535b64\") "
Oct 31 00:29:48.029509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.029395    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume" pod "0165dc8b-2e01-4dba-a5bc-04c847535b64" (UID: "0165dc8b-2e01-4dba-a5bc-04c847535b64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:48.030198 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.030173    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxf9\" (UniqueName: \"kubernetes.io/projected/bc629a17-3d53-4c71-8ca9-8d140ee3618a-kube-api-access-zkxf9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:48.030316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.030203    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0165dc8b-2e01-4dba-a5bc-04c847535b64-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:48.033305 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.033267    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5" pod "0165dc8b-2e01-4dba-a5bc-04c847535b64" (UID: "0165dc8b-2e01-4dba-a5bc-04c847535b64"). InnerVolumeSpecName "kube-api-access-8z4l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:48.135722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.131733    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4l5\" (UniqueName: \"kubernetes.io/projected/0165dc8b-2e01-4dba-a5bc-04c847535b64-kube-api-access-8z4l5\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:48.293249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.291622    5078 generic.go:356] "Generic (PLEG): container finished" podID="bc629a17-3d53-4c71-8ca9-8d140ee3618a" containerID="a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0" exitCode=0
Oct 31 00:29:48.293249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.292456    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"
Oct 31 00:29:48.303700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.303670    5078 generic.go:356] "Generic (PLEG): container finished" podID="0165dc8b-2e01-4dba-a5bc-04c847535b64" containerID="da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818" exitCode=0
Oct 31 00:29:48.304143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.303819    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"
Oct 31 00:29:48.308291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.308264    5078 generic.go:356] "Generic (PLEG): container finished" podID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerID="e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7" exitCode=0
Oct 31 00:29:48.308523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.308503    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313609    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q" event={"ID":"bc629a17-3d53-4c71-8ca9-8d140ee3618a","Type":"ContainerDied","Data":"a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0"}
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313649    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q" event={"ID":"bc629a17-3d53-4c71-8ca9-8d140ee3618a","Type":"ContainerDied","Data":"53353b3b94530bcc07d360bf396abd0fda9e1322e9a16e3bc52bb1c5b1789ee5"}
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313666    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r" event={"ID":"0165dc8b-2e01-4dba-a5bc-04c847535b64","Type":"ContainerDied","Data":"da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818"}
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313713    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r" event={"ID":"0165dc8b-2e01-4dba-a5bc-04c847535b64","Type":"ContainerDied","Data":"869fd8495d89c0ab43cf240d52ea306784a21e671d7c23349d5882d8d5b4934a"}
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313745    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" event={"ID":"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f","Type":"ContainerDied","Data":"e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7"}
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313762    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd" event={"ID":"75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f","Type":"ContainerDied","Data":"31d62ec50e4bd1048fb583f11c5797c051252b7b5eeb1f1c15284c4f7cc2da5d"}
Oct 31 00:29:48.314632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.313785    5078 scope.go:122] "RemoveContainer" containerID="a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0"
Oct 31 00:29:48.337302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.337276    5078 scope.go:122] "RemoveContainer" containerID="a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0"
Oct 31 00:29:48.337573 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:48.337541    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0\": container with ID starting with a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0 not found: ID does not exist" containerID="a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0"
Oct 31 00:29:48.337667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.337577    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0"} err="failed to get container status \"a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0\": rpc error: code = NotFound desc = could not find container \"a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0\": container with ID starting with a05fac37144a44d30cda489a5bd3a864fe95a9190992a8d29ca92726221f45b0 not found: ID does not exist"
Oct 31 00:29:48.337667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.337600    5078 scope.go:122] "RemoveContainer" containerID="da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818"
Oct 31 00:29:48.342390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.342366    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"]
Oct 31 00:29:48.346888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.346782    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d9875-7dc4d4cf7-s8w8q"]
Oct 31 00:29:48.355437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.355417    5078 scope.go:122] "RemoveContainer" containerID="da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818"
Oct 31 00:29:48.355850 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:48.355826    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818\": container with ID starting with da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818 not found: ID does not exist" containerID="da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818"
Oct 31 00:29:48.355930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.355859    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818"} err="failed to get container status \"da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818\": rpc error: code = NotFound desc = could not find container \"da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818\": container with ID starting with da4be1197847ab3a7eeffb51cbed4b239dd1378c7b560ccf96eeee511219f818 not found: ID does not exist"
Oct 31 00:29:48.355930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.355882    5078 scope.go:122] "RemoveContainer" containerID="e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7"
Oct 31 00:29:48.366637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.366612    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"]
Oct 31 00:29:48.370627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.370602    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/premium-w1-4c360e-kserve-74fd457656-4pqtd"]
Oct 31 00:29:48.370972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.370960    5078 scope.go:122] "RemoveContainer" containerID="e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7"
Oct 31 00:29:48.371222 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:48.371202    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7\": container with ID starting with e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7 not found: ID does not exist" containerID="e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7"
Oct 31 00:29:48.371269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.371229    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7"} err="failed to get container status \"e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7\": rpc error: code = NotFound desc = could not find container \"e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7\": container with ID starting with e115cb3f06dfc19668010122a8b5a460388ccb653f13b95de5912c36fecd0ce7 not found: ID does not exist"
Oct 31 00:29:48.379876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.379764    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"]
Oct 31 00:29:48.385498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.385479    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d9875-6c57b5948c-hc29r"]
Oct 31 00:29:48.861494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.858830    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" containerID="cri-o://cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1" gracePeriod=43
Oct 31 00:29:48.878917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:48.877632    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": EOF"
Oct 31 00:29:49.207849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.207825    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:29:49.260940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.260912    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home\") pod \"869264e1-101e-4adb-b8f6-17bfa992b217\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") "
Oct 31 00:29:49.261118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261043    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache\") pod \"869264e1-101e-4adb-b8f6-17bfa992b217\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") "
Oct 31 00:29:49.261191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261177    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs\" (UniqueName: \"kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs\") pod \"869264e1-101e-4adb-b8f6-17bfa992b217\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") "
Oct 31 00:29:49.261258 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261187    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home" pod "869264e1-101e-4adb-b8f6-17bfa992b217" (UID: "869264e1-101e-4adb-b8f6-17bfa992b217"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:49.261387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261358    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache" pod "869264e1-101e-4adb-b8f6-17bfa992b217" (UID: "869264e1-101e-4adb-b8f6-17bfa992b217"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:49.261655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261631    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm\" (UniqueName: \"kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm\") pod \"869264e1-101e-4adb-b8f6-17bfa992b217\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") "
Oct 31 00:29:49.261810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261701    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir\") pod \"869264e1-101e-4adb-b8f6-17bfa992b217\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") "
Oct 31 00:29:49.261810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.261745    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm\") pod \"869264e1-101e-4adb-b8f6-17bfa992b217\" (UID: \"869264e1-101e-4adb-b8f6-17bfa992b217\") "
Oct 31 00:29:49.262346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.262197    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir" pod "869264e1-101e-4adb-b8f6-17bfa992b217" (UID: "869264e1-101e-4adb-b8f6-17bfa992b217"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:49.263120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.263101    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:49.263287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.263273    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:49.263389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.263378    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:49.264768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.263924    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs" pod "869264e1-101e-4adb-b8f6-17bfa992b217" (UID: "869264e1-101e-4adb-b8f6-17bfa992b217"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:49.265860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.265827    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm" pod "869264e1-101e-4adb-b8f6-17bfa992b217" (UID: "869264e1-101e-4adb-b8f6-17bfa992b217"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:49.267958 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.267930    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm" pod "869264e1-101e-4adb-b8f6-17bfa992b217" (UID: "869264e1-101e-4adb-b8f6-17bfa992b217"). InnerVolumeSpecName "kube-api-access-v8gkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:49.319918 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.319882    5078 generic.go:356] "Generic (PLEG): container finished" podID="869264e1-101e-4adb-b8f6-17bfa992b217" containerID="cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1" exitCode=0
Oct 31 00:29:49.320046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.319932    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"
Oct 31 00:29:49.320046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.319964    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" event={"ID":"869264e1-101e-4adb-b8f6-17bfa992b217","Type":"ContainerDied","Data":"cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1"}
Oct 31 00:29:49.320046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.320006    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb" event={"ID":"869264e1-101e-4adb-b8f6-17bfa992b217","Type":"ContainerDied","Data":"7820824ab8f94a2229586d57f96b261e3fb66f7e1f63a6509726f9d0293ed67d"}
Oct 31 00:29:49.320046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.320023    5078 scope.go:122] "RemoveContainer" containerID="cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1"
Oct 31 00:29:49.332919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.332903    5078 scope.go:122] "RemoveContainer" containerID="cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1"
Oct 31 00:29:49.333193 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:49.333174    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1\": container with ID starting with cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1 not found: ID does not exist" containerID="cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1"
Oct 31 00:29:49.333265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.333202    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1"} err="failed to get container status \"cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1\": rpc error: code = NotFound desc = could not find container \"cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1\": container with ID starting with cd76470d983037f9a2c1ab46a42a95adcfc7e926f036dcaaddbb1e900146fde1 not found: ID does not exist"
Oct 31 00:29:49.357581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.357547    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"]
Oct 31 00:29:49.364639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.364613    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-2-w1-4c360e-kserve-759c984544-2stvb"]
Oct 31 00:29:49.364900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.364878    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869264e1-101e-4adb-b8f6-17bfa992b217-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:49.364952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.364909    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gkm\" (UniqueName: \"kubernetes.io/projected/869264e1-101e-4adb-b8f6-17bfa992b217-kube-api-access-v8gkm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:49.364952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.364926    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869264e1-101e-4adb-b8f6-17bfa992b217-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:49.453364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.453284    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"]
Oct 31 00:29:49.453591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:49.453567    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy" containerID="cri-o://253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413" gracePeriod=30
Oct 31 00:29:50.197555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:50.197515    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0165dc8b-2e01-4dba-a5bc-04c847535b64" path="/var/lib/kubelet/pods/0165dc8b-2e01-4dba-a5bc-04c847535b64/volumes"
Oct 31 00:29:50.198003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:50.197988    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" path="/var/lib/kubelet/pods/75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f/volumes"
Oct 31 00:29:50.198422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:50.198408    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" path="/var/lib/kubelet/pods/869264e1-101e-4adb-b8f6-17bfa992b217/volumes"
Oct 31 00:29:50.199146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:50.199130    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc629a17-3d53-4c71-8ca9-8d140ee3618a" path="/var/lib/kubelet/pods/bc629a17-3d53-4c71-8ca9-8d140ee3618a/volumes"
Oct 31 00:29:50.434966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:50.434931    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"]
Oct 31 00:29:50.435201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:50.435176    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy" containerID="cri-o://6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92" gracePeriod=30
Oct 31 00:29:53.788319 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:53.788284    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"]
Oct 31 00:29:53.788815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:53.788552    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" containerID="cri-o://593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083" gracePeriod=30
Oct 31 00:29:53.946771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:53.946727    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"]
Oct 31 00:29:53.947030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:53.946999    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" containerID="cri-o://0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18" gracePeriod=30
Oct 31 00:29:53.999488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:53.999400    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"]
Oct 31 00:29:53.999745 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:53.999715    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy" containerID="cri-o://ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491" gracePeriod=30
Oct 31 00:29:54.832288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:54.832259    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:29:55.016289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016256    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016295    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016328    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016365    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016569    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016680    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016689    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:55.016754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016706    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:55.016754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016713    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.016968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016891    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:55.016968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.016867    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.017122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.017026    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:55.017183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.017091    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:55.017183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.017095    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo\") pod \"aa8de2cb-4f63-4998-93a9-aff00d53183e\" (UID: \"aa8de2cb-4f63-4998-93a9-aff00d53183e\") "
Oct 31 00:29:55.018271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.018245    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.018400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.018275    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8de2cb-4f63-4998-93a9-aff00d53183e-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.018400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.018289    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.018400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.018304    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.018400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.018321    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.019352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.019326    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:55.019352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.019344    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:55.019528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.019455    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:55.019574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.019532    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8" pod "aa8de2cb-4f63-4998-93a9-aff00d53183e" (UID: "aa8de2cb-4f63-4998-93a9-aff00d53183e"). InnerVolumeSpecName "kube-api-access-9rmd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:55.120301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.120224    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rmd8\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-kube-api-access-9rmd8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.120301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.120260    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.120301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.120277    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.120301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.120290    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8de2cb-4f63-4998-93a9-aff00d53183e-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:55.387849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.387810    5078 generic.go:356] "Generic (PLEG): container finished" podID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerID="253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413" exitCode=0
Oct 31 00:29:55.388005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.387897    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"
Oct 31 00:29:55.388005 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.387893    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" event={"ID":"aa8de2cb-4f63-4998-93a9-aff00d53183e","Type":"ContainerDied","Data":"253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413"}
Oct 31 00:29:55.388101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.388034    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q" event={"ID":"aa8de2cb-4f63-4998-93a9-aff00d53183e","Type":"ContainerDied","Data":"d382f98c0b597c4281d978892d1a874df9c169480b1f03be0833c29c6383e38e"}
Oct 31 00:29:55.388101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.388067    5078 scope.go:122] "RemoveContainer" containerID="253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413"
Oct 31 00:29:55.401810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.401777    5078 scope.go:122] "RemoveContainer" containerID="253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413"
Oct 31 00:29:55.402180 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:55.402154    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413\": container with ID starting with 253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413 not found: ID does not exist" containerID="253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413"
Oct 31 00:29:55.402271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.402193    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413"} err="failed to get container status \"253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413\": rpc error: code = NotFound desc = could not find container \"253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413\": container with ID starting with 253cfe7c0a71a811c67962559839a80ad8dc10136f976a051778bd6624b04413 not found: ID does not exist"
Oct 31 00:29:55.437727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.437667    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"]
Oct 31 00:29:55.468521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:55.468499    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-7n49q"]
Oct 31 00:29:56.145570 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.145534    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:29:56.154984 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.154956    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:29:56.155111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.155005    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:29:56.221735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.221659    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" path="/var/lib/kubelet/pods/aa8de2cb-4f63-4998-93a9-aff00d53183e/volumes"
Oct 31 00:29:56.340717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.340684    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.340932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.340728    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.340932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.340802    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.340932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.340852    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.341126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.340994    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.341126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341053    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.341239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341132    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:56.341239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341143    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.341364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341275    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:56.341364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341287    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:56.341364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.341364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341357    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo\") pod \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\" (UID: \"3d0fcca2-d2a7-4b96-8271-151d57fa2587\") "
Oct 31 00:29:56.341635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341610    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:56.341706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.341542    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:56.342814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.342590    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.342814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.342617    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.342814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.342651    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.342814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.342666    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.342814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.342682    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.343690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.343659    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:56.343690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.343664    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:56.344200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.344174    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "kube-api-access-x79z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:56.344341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.344324    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo" pod "3d0fcca2-d2a7-4b96-8271-151d57fa2587" (UID: "3d0fcca2-d2a7-4b96-8271-151d57fa2587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:56.345888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.345858    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:29:56.346000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.345909    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:29:56.400657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.400624    5078 generic.go:356] "Generic (PLEG): container finished" podID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerID="6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92" exitCode=0
Oct 31 00:29:56.400833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.400669    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" event={"ID":"3d0fcca2-d2a7-4b96-8271-151d57fa2587","Type":"ContainerDied","Data":"6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92"}
Oct 31 00:29:56.400833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.400737    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd" event={"ID":"3d0fcca2-d2a7-4b96-8271-151d57fa2587","Type":"ContainerDied","Data":"5f56eb583ec9eed418c85b3872e0d2901ba2b22846eb476850bbf591319c40a0"}
Oct 31 00:29:56.400833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.400681    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"
Oct 31 00:29:56.400833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.400767    5078 scope.go:122] "RemoveContainer" containerID="6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92"
Oct 31 00:29:56.419148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.419127    5078 scope.go:122] "RemoveContainer" containerID="6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92"
Oct 31 00:29:56.419525 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:56.419500    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92\": container with ID starting with 6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92 not found: ID does not exist" containerID="6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92"
Oct 31 00:29:56.419619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.419538    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92"} err="failed to get container status \"6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92\": rpc error: code = NotFound desc = could not find container \"6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92\": container with ID starting with 6b32c503ccfe8996f5ae7e5cbbd06053190fbbb77420ea941a260861dc373e92 not found: ID does not exist"
Oct 31 00:29:56.439004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.438970    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"]
Oct 31 00:29:56.443801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.443774    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.443801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.443799    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.443979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.443808    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.443979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.443820    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79z6\" (UniqueName: \"kubernetes.io/projected/3d0fcca2-d2a7-4b96-8271-151d57fa2587-kube-api-access-x79z6\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.446773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.446750    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-xf5pd"]
Oct 31 00:29:56.581300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.581218    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"]
Oct 31 00:29:56.581534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:56.581504    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr" podUID="381d0146-cbfd-4a00-bdff-7947f9f2a459" containerName="maas-api" containerID="cri-o://d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7" gracePeriod=30
Oct 31 00:29:57.047380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.047348    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:29:57.086775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.085941    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls\") pod \"381d0146-cbfd-4a00-bdff-7947f9f2a459\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") "
Oct 31 00:29:57.086775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.085992    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca\") pod \"381d0146-cbfd-4a00-bdff-7947f9f2a459\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") "
Oct 31 00:29:57.086775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.086108    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs\") pod \"381d0146-cbfd-4a00-bdff-7947f9f2a459\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") "
Oct 31 00:29:57.086775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.086232    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29\" (UniqueName: \"kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29\") pod \"381d0146-cbfd-4a00-bdff-7947f9f2a459\" (UID: \"381d0146-cbfd-4a00-bdff-7947f9f2a459\") "
Oct 31 00:29:57.088508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.087808    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca" pod "381d0146-cbfd-4a00-bdff-7947f9f2a459" (UID: "381d0146-cbfd-4a00-bdff-7947f9f2a459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:57.099062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.097049    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29" pod "381d0146-cbfd-4a00-bdff-7947f9f2a459" (UID: "381d0146-cbfd-4a00-bdff-7947f9f2a459"). InnerVolumeSpecName "kube-api-access-clb29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:57.101290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.101258    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs" pod "381d0146-cbfd-4a00-bdff-7947f9f2a459" (UID: "381d0146-cbfd-4a00-bdff-7947f9f2a459"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:57.109978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.109951    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls" pod "381d0146-cbfd-4a00-bdff-7947f9f2a459" (UID: "381d0146-cbfd-4a00-bdff-7947f9f2a459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:57.188264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.188231    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clb29\" (UniqueName: \"kubernetes.io/projected/381d0146-cbfd-4a00-bdff-7947f9f2a459-kube-api-access-clb29\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:57.188264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.188268    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:57.188776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.188284    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d0146-cbfd-4a00-bdff-7947f9f2a459-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:57.188776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.188297    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381d0146-cbfd-4a00-bdff-7947f9f2a459-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:57.410066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.410019    5078 generic.go:356] "Generic (PLEG): container finished" podID="381d0146-cbfd-4a00-bdff-7947f9f2a459" containerID="d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7" exitCode=0
Oct 31 00:29:57.410247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.410078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"
Oct 31 00:29:57.410247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.410111    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr" event={"ID":"381d0146-cbfd-4a00-bdff-7947f9f2a459","Type":"ContainerDied","Data":"d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7"}
Oct 31 00:29:57.410247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.410172    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr" event={"ID":"381d0146-cbfd-4a00-bdff-7947f9f2a459","Type":"ContainerDied","Data":"1e81822654ecedb1b79badb81015de46db6dd941d32dccefd4c1d62ee4193ae0"}
Oct 31 00:29:57.410247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.410199    5078 scope.go:122] "RemoveContainer" containerID="d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7"
Oct 31 00:29:57.424855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.424815    5078 scope.go:122] "RemoveContainer" containerID="d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7"
Oct 31 00:29:57.425150 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:57.425125    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7\": container with ID starting with d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7 not found: ID does not exist" containerID="d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7"
Oct 31 00:29:57.425256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.425160    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7"} err="failed to get container status \"d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7\": rpc error: code = NotFound desc = could not find container \"d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7\": container with ID starting with d310f51b982bc3dd06fd5a40f8f7b4b158f60933f6c46d03804100e7af2720c7 not found: ID does not exist"
Oct 31 00:29:57.442103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.442078    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"]
Oct 31 00:29:57.447201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:57.447174    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d898f3-cf9f95f96-f5nnr"]
Oct 31 00:29:58.198837 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:58.198806    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381d0146-cbfd-4a00-bdff-7947f9f2a459" path="/var/lib/kubelet/pods/381d0146-cbfd-4a00-bdff-7947f9f2a459/volumes"
Oct 31 00:29:58.199353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:58.199338    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" path="/var/lib/kubelet/pods/3d0fcca2-d2a7-4b96-8271-151d57fa2587/volumes"
Oct 31 00:29:59.039231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.039160    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"]
Oct 31 00:29:59.044739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.044712    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ede3ba-ps7nr"]
Oct 31 00:29:59.165528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.164765    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219040    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219283    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219335    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219555    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219614    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219690    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219861    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219911    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.221081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.219940    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r\") pod \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\" (UID: \"22f1a149-d6c7-4c5f-97c1-639866c3b6f5\") "
Oct 31 00:29:59.222037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.221820    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.222121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.222034    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.224648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.224442    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:59.224807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.224636    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.224807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.224733    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.228215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.228180    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:59.228585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.228551    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "kube-api-access-zrz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:59.230207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.230174    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.251642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.249696    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo" pod "22f1a149-d6c7-4c5f-97c1-639866c3b6f5" (UID: "22f1a149-d6c7-4c5f-97c1-639866c3b6f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323667    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323703    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323722    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323739    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrz4r\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-kube-api-access-zrz4r\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323754    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323767    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323781    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323794    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.327500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.323807    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f1a149-d6c7-4c5f-97c1-639866c3b6f5-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.331055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.331027    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"]
Oct 31 00:29:59.350998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.348293    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"]
Oct 31 00:29:59.418267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.418237    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:29:59.420283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.420256    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:29:59.439352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.439326    5078 generic.go:356] "Generic (PLEG): container finished" podID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerID="593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083" exitCode=0
Oct 31 00:29:59.439556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.439388    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"
Oct 31 00:29:59.439556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.439413    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" event={"ID":"22f1a149-d6c7-4c5f-97c1-639866c3b6f5","Type":"ContainerDied","Data":"593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083"}
Oct 31 00:29:59.439556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.439517    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz" event={"ID":"22f1a149-d6c7-4c5f-97c1-639866c3b6f5","Type":"ContainerDied","Data":"56798db62f88b7b12f1cc236f2d2ad4c45639d226b2b7d03382b4181db18b957"}
Oct 31 00:29:59.439556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.439549    5078 scope.go:122] "RemoveContainer" containerID="593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083"
Oct 31 00:29:59.440994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.440972    5078 generic.go:356] "Generic (PLEG): container finished" podID="8b949d93-a549-4046-8375-15e54524a8da" containerID="0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18" exitCode=0
Oct 31 00:29:59.441119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.441080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" event={"ID":"8b949d93-a549-4046-8375-15e54524a8da","Type":"ContainerDied","Data":"0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18"}
Oct 31 00:29:59.441205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.441091    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"
Oct 31 00:29:59.441295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.441122    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v" event={"ID":"8b949d93-a549-4046-8375-15e54524a8da","Type":"ContainerDied","Data":"c365231cb7c45b5784a684fcf54100429aa7c90faa2c6339e629353323d2da37"}
Oct 31 00:29:59.442800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.442783    5078 generic.go:356] "Generic (PLEG): container finished" podID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerID="ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491" exitCode=0
Oct 31 00:29:59.442966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.442807    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" event={"ID":"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7","Type":"ContainerDied","Data":"ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491"}
Oct 31 00:29:59.443103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.442999    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m" event={"ID":"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7","Type":"ContainerDied","Data":"65f27afad2a26a454e28e2a56a9d768cfeefe05f5ecb0d3ec6ae2c8d72490aff"}
Oct 31 00:29:59.443103 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.442830    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"
Oct 31 00:29:59.457563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.457542    5078 scope.go:122] "RemoveContainer" containerID="9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"
Oct 31 00:29:59.493959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.493887    5078 scope.go:122] "RemoveContainer" containerID="593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083"
Oct 31 00:29:59.494271 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:59.494245    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083\": container with ID starting with 593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083 not found: ID does not exist" containerID="593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083"
Oct 31 00:29:59.494375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.494284    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083"} err="failed to get container status \"593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083\": rpc error: code = NotFound desc = could not find container \"593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083\": container with ID starting with 593a21b4ecb89d4a10d7349edd933670a99e198ab34f14d6642c7af23123b083 not found: ID does not exist"
Oct 31 00:29:59.494375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.494309    5078 scope.go:122] "RemoveContainer" containerID="9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"
Oct 31 00:29:59.494689 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:59.494664    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c\": container with ID starting with 9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c not found: ID does not exist" containerID="9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"
Oct 31 00:29:59.494755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.494702    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c"} err="failed to get container status \"9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c\": rpc error: code = NotFound desc = could not find container \"9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c\": container with ID starting with 9315697a97111b5a664e8a7a57f6bfab791aa3ec56abf48c0b6ed163cfc1d37c not found: ID does not exist"
Oct 31 00:29:59.494755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.494729    5078 scope.go:122] "RemoveContainer" containerID="0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18"
Oct 31 00:29:59.513147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.513122    5078 scope.go:122] "RemoveContainer" containerID="0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18"
Oct 31 00:29:59.513571 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:59.513530    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18\": container with ID starting with 0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18 not found: ID does not exist" containerID="0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18"
Oct 31 00:29:59.513720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.513581    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18"} err="failed to get container status \"0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18\": rpc error: code = NotFound desc = could not find container \"0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18\": container with ID starting with 0a9ffe56baac2f039da285a85937aff79e729f613944439f0bb88f6b66ef1b18 not found: ID does not exist"
Oct 31 00:29:59.513720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.513605    5078 scope.go:122] "RemoveContainer" containerID="ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491"
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.528892    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.528951    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529004    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529059    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529103    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529136    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529237    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529276    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529311    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529369    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529401    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529439    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529514    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529585    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529625    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529682    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529812    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z\") pod \"8b949d93-a549-4046-8375-15e54524a8da\" (UID: \"8b949d93-a549-4046-8375-15e54524a8da\") "
Oct 31 00:29:59.531706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.529906    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token\") pod \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\" (UID: \"d14e217a-7ac6-40b8-9f4f-a17cb2039ef7\") "
Oct 31 00:29:59.533588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.533334    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.534800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.533859    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:59.534800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.534290    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.534800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.534544    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.534800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.534728    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.535093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.534913    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.535155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.535103    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.535379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.535284    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.536156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.535521    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.536156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.535748    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.536156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.535845    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:59.536156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.535931    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:59.538908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.538839    5078 scope.go:122] "RemoveContainer" containerID="ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491"
Oct 31 00:29:59.539039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.539013    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "kube-api-access-wqm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:59.539516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.539373    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:59.539516 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:29:59.539395    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491\": container with ID starting with ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491 not found: ID does not exist" containerID="ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491"
Oct 31 00:29:59.539516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.539428    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491"} err="failed to get container status \"ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491\": rpc error: code = NotFound desc = could not find container \"ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491\": container with ID starting with ca9eae58a3441d9f00764b01dea3b4dbf02fb1aebb8685b771764da94b517491 not found: ID does not exist"
Oct 31 00:29:59.539888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.539861    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:59.539973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.539930    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "kube-api-access-mkvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:59.540423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540399    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token" pod "8b949d93-a549-4046-8375-15e54524a8da" (UID: "8b949d93-a549-4046-8375-15e54524a8da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:59.540787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540766    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.540861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540797    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b949d93-a549-4046-8375-15e54524a8da-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.540861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540814    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.540861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540829    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.540861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540842    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.540861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540857    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540870    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540884    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b949d93-a549-4046-8375-15e54524a8da-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540906    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540919    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540932    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540945    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540960    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540972    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b949d93-a549-4046-8375-15e54524a8da-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540986    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqm8z\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-kube-api-access-wqm8z\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.540999    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b949d93-a549-4046-8375-15e54524a8da-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.541012    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkvjv\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-kube-api-access-mkvjv\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.541420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.541187    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token" pod "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" (UID: "d14e217a-7ac6-40b8-9f4f-a17cb2039ef7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:59.552575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.552553    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"]
Oct 31 00:29:59.562775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.562756    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-2h6sz"]
Oct 31 00:29:59.642252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.642228    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:59.786932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.786895    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"]
Oct 31 00:29:59.795615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.795582    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-brn2v"]
Oct 31 00:29:59.812271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.812231    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"]
Oct 31 00:29:59.816031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:29:59.816000    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-jmx7m"]
Oct 31 00:30:00.179727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.179693    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"]
Oct 31 00:30:00.180433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180416    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0165dc8b-2e01-4dba-a5bc-04c847535b64" containerName="payload-pre-processing"
Oct 31 00:30:00.180501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180435    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0165dc8b-2e01-4dba-a5bc-04c847535b64" containerName="payload-pre-processing"
Oct 31 00:30:00.180501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180492    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180502    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180516    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180521    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180531    5078 cpu_manager.go:414] "containerMap: removing container" podUID="381d0146-cbfd-4a00-bdff-7947f9f2a459" containerName="maas-api"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180535    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381d0146-cbfd-4a00-bdff-7947f9f2a459" containerName="maas-api"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180544    5078 cpu_manager.go:414] "containerMap: removing container" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180548    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180557    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180562    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180572    5078 cpu_manager.go:414] "containerMap: removing container" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180576    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy"
Oct 31 00:30:00.180577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180581    5078 cpu_manager.go:414] "containerMap: removing container" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180585    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180597    5078 cpu_manager.go:414] "containerMap: removing container" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180601    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180607    5078 cpu_manager.go:414] "containerMap: removing container" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180611    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180618    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c0429edc-ac06-4abe-8811-31047ba93cf4" containerName="payload-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180622    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0429edc-ac06-4abe-8811-31047ba93cf4" containerName="payload-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180634    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180638    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180646    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180650    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180659    5078 cpu_manager.go:414] "containerMap: removing container" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180663    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180669    5078 cpu_manager.go:414] "containerMap: removing container" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180673    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180679    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180684    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180694    5078 cpu_manager.go:414] "containerMap: removing container" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180698    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180712    5078 cpu_manager.go:414] "containerMap: removing container" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180716    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180732    5078 cpu_manager.go:414] "containerMap: removing container" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180736    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180745    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180749    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180757    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180761    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180769    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bc629a17-3d53-4c71-8ca9-8d140ee3618a" containerName="payload-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180774    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc629a17-3d53-4c71-8ca9-8d140ee3618a" containerName="payload-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180806    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180812    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180819    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180824    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180831    5078 cpu_manager.go:414] "containerMap: removing container" podUID="55fa4ee3-9bfd-43ac-97c6-9448a4635262" containerName="payload-pre-processing"
Oct 31 00:30:00.181020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.180836    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fa4ee3-9bfd-43ac-97c6-9448a4635262" containerName="payload-pre-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181055    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8de2cb-4f63-4998-93a9-aff00d53183e" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181075    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0fcca2-d2a7-4b96-8271-151d57fa2587" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181084    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181093    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="010f4cfc-84d4-4b18-a9ff-289488077e9b" containerName="maas-api"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181104    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181112    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181120    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0429edc-ac06-4abe-8811-31047ba93cf4" containerName="payload-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181131    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75e181d-6850-4b90-a34a-3a1d098a6f53" containerName="payload-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181142    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181151    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="869264e1-101e-4adb-b8f6-17bfa992b217" containerName="main"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181158    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="325f8eed-03c3-4f20-a72a-7753e560cb45" containerName="payload-pre-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181169    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181177    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf6aa4-c426-47d8-a8b2-fe5d95c1e22f" containerName="main"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181191    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181199    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a603b11-c6d9-4c14-932f-c1e3d1454bea" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181210    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a23b543-84bf-4a68-9e80-414b3fdfc4ea" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181217    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc629a17-3d53-4c71-8ca9-8d140ee3618a" containerName="payload-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181227    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b20778e-6adf-46f7-ae50-b389aa95f36c" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181236    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4de92e-ac35-496d-9c11-297373884443" containerName="payload-pre-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181243    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fa4ee3-9bfd-43ac-97c6-9448a4635262" containerName="payload-pre-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181254    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0165dc8b-2e01-4dba-a5bc-04c847535b64" containerName="payload-pre-processing"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181266    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b949d93-a549-4046-8375-15e54524a8da" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181276    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0038939b-91e8-4b4c-848f-2da60082ff8e" containerName="istio-proxy"
Oct 31 00:30:00.182252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.181287    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="381d0146-cbfd-4a00-bdff-7947f9f2a459" containerName="maas-api"
Oct 31 00:30:00.186377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.186357    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"]
Oct 31 00:30:00.186526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.186509    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.187127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.187109    5078 cpu_manager.go:414] "containerMap: removing container" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy"
Oct 31 00:30:00.187127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.187127    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy"
Oct 31 00:30:00.187281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.187228    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="86417f75-420a-4ef6-84ee-1d36bd8632ca" containerName="istio-proxy"
Oct 31 00:30:00.187281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.187248    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="86073226-02be-4443-b886-654281e0a89a" containerName="istio-proxy"
Oct 31 00:30:00.192049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.191611    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.208316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.206656    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1563cf96-6d7c-4dc9-8911-710b77014179" path="/var/lib/kubelet/pods/1563cf96-6d7c-4dc9-8911-710b77014179/volumes"
Oct 31 00:30:00.208316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.207534    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f1a149-d6c7-4c5f-97c1-639866c3b6f5" path="/var/lib/kubelet/pods/22f1a149-d6c7-4c5f-97c1-639866c3b6f5/volumes"
Oct 31 00:30:00.208456 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.208352    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b949d93-a549-4046-8375-15e54524a8da" path="/var/lib/kubelet/pods/8b949d93-a549-4046-8375-15e54524a8da/volumes"
Oct 31 00:30:00.218526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.211064    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14e217a-7ac6-40b8-9f4f-a17cb2039ef7" path="/var/lib/kubelet/pods/d14e217a-7ac6-40b8-9f4f-a17cb2039ef7/volumes"
Oct 31 00:30:00.232520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.223685    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:30:00.241793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.241132    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"]
Oct 31 00:30:00.251798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.251766    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.252407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.252377    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.252561 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.252425    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9d2h\" (UniqueName: \"kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.252882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.252857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqlp\" (UniqueName: \"kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.258609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.258489    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"]
Oct 31 00:30:00.264427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.264403    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"]
Oct 31 00:30:00.277104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.277082    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"]
Oct 31 00:30:00.277221 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.277186    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.280647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.280621    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"]
Oct 31 00:30:00.288601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.288403    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"]
Oct 31 00:30:00.288726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.288628    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.313691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.313627    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"]
Oct 31 00:30:00.321388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.320142    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.352770    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"]
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354180    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354259    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29z9\" (UniqueName: \"kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkvr\" (UniqueName: \"kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354548    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354590    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354616    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354648    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9d2h\" (UniqueName: \"kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354716    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcsf\" (UniqueName: \"kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354753    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqlp\" (UniqueName: \"kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.354779    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.355713    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.356369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.382044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.378755    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqlp\" (UniqueName: \"kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.394535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.384622    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9d2h\" (UniqueName: \"kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.456018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.455926    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkvr\" (UniqueName: \"kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.456200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456013    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.456200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456084    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcsf\" (UniqueName: \"kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.456200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456183    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.456379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456262    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.456379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456325    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29z9\" (UniqueName: \"kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.456886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456867    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.456975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.456965    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.457074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.457054    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.488362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.488335    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkvr\" (UniqueName: \"kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr\") pod \"maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.488594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.488335    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29z9\" (UniqueName: \"kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.492784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.492758    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcsf\" (UniqueName: \"kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.513783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.513753    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:00.546031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.546004    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:00.611282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.610994    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:00.627074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.626841    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:00.641698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.641454    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:00.818263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.816075    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"]
Oct 31 00:30:00.821266 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:00.821235    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223c462f_c0fc_41ad_b18d_60b2dcb4f4ca.slice/crio-974b85ba056f0f156f277a38785a415fef07199f6d59ab7d24afb1b230fe4b68 WatchSource:0}: Error finding container 974b85ba056f0f156f277a38785a415fef07199f6d59ab7d24afb1b230fe4b68: Status 404 returned error can't find the container with id 974b85ba056f0f156f277a38785a415fef07199f6d59ab7d24afb1b230fe4b68
Oct 31 00:30:00.863157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.863100    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"]
Oct 31 00:30:00.945147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.945117    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"]
Oct 31 00:30:00.945293 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:00.945268    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfeda2b_c240_4259_8ac0_0cffa34992de.slice/crio-1c4904030346b9e3afef96f19a34f850b238b38bf47c3c40412677bccbeac58c WatchSource:0}: Error finding container 1c4904030346b9e3afef96f19a34f850b238b38bf47c3c40412677bccbeac58c: Status 404 returned error can't find the container with id 1c4904030346b9e3afef96f19a34f850b238b38bf47c3c40412677bccbeac58c
Oct 31 00:30:00.953074 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:00.953051    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04a4cd4_2941_4cd0_a378_7d590d70223f.slice/crio-427979a102cbf4d5092bdb4810fd1913cb48ed78dbbc2dd5713f2c462b402aa7 WatchSource:0}: Error finding container 427979a102cbf4d5092bdb4810fd1913cb48ed78dbbc2dd5713f2c462b402aa7: Status 404 returned error can't find the container with id 427979a102cbf4d5092bdb4810fd1913cb48ed78dbbc2dd5713f2c462b402aa7
Oct 31 00:30:00.955690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.955669    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"]
Oct 31 00:30:00.989646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:00.989617    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"]
Oct 31 00:30:01.075981 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:01.075945    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc181ef9_ba0b_41fc_9933_4e113f546ce8.slice/crio-2d9e9b801a465a5d1b348eb3d8286319df61f9245a8a755d4d2f33f12d780373 WatchSource:0}: Error finding container 2d9e9b801a465a5d1b348eb3d8286319df61f9245a8a755d4d2f33f12d780373: Status 404 returned error can't find the container with id 2d9e9b801a465a5d1b348eb3d8286319df61f9245a8a755d4d2f33f12d780373
Oct 31 00:30:01.466707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.466676    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerStarted","Data":"1c4904030346b9e3afef96f19a34f850b238b38bf47c3c40412677bccbeac58c"}
Oct 31 00:30:01.468379 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.468350    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerStarted","Data":"78e8da45caf6cf507a1d7f396f8f9561b91ae98314aa0846e16ab7348550b3db"}
Oct 31 00:30:01.468532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.468402    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerStarted","Data":"974b85ba056f0f156f277a38785a415fef07199f6d59ab7d24afb1b230fe4b68"}
Oct 31 00:30:01.470431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.470407    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerStarted","Data":"0f8317a9acfc135ca42e279ecbb331c694597cde711f2683683de46e612da92b"}
Oct 31 00:30:01.470672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.470444    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerStarted","Data":"789c2df012fabd05a3af4b1a2ee853d33c00d4e88f283c52457b912407dba2c7"}
Oct 31 00:30:01.473983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.472774    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerStarted","Data":"2d9e9b801a465a5d1b348eb3d8286319df61f9245a8a755d4d2f33f12d780373"}
Oct 31 00:30:01.475627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.475605    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" event={"ID":"a04a4cd4-2941-4cd0-a378-7d590d70223f","Type":"ContainerStarted","Data":"427979a102cbf4d5092bdb4810fd1913cb48ed78dbbc2dd5713f2c462b402aa7"}
Oct 31 00:30:01.491202 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.491147    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" podStartSLOduration=1.491127355 podStartE2EDuration="1.491127355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:01.488567679 +0000 UTC m=+2169.434938397" watchObservedRunningTime="2026-10-31 00:30:01.491127355 +0000 UTC m=+2169.437498073"
Oct 31 00:30:01.513736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:01.513682    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" podStartSLOduration=1.513669057 podStartE2EDuration="1.513669057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:01.509409088 +0000 UTC m=+2169.455779806" watchObservedRunningTime="2026-10-31 00:30:01.513669057 +0000 UTC m=+2169.460039774"
Oct 31 00:30:02.486093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.486053    5078 generic.go:356] "Generic (PLEG): container finished" podID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerID="51445a9178e8369be2295eff4ac03dd6ad309c52756cd51dde99b682ae6f1de1" exitCode=60
Oct 31 00:30:02.486615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.486120    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerDied","Data":"51445a9178e8369be2295eff4ac03dd6ad309c52756cd51dde99b682ae6f1de1"}
Oct 31 00:30:02.486681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.486621    5078 scope.go:122] "RemoveContainer" containerID="51445a9178e8369be2295eff4ac03dd6ad309c52756cd51dde99b682ae6f1de1"
Oct 31 00:30:02.489259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.488783    5078 generic.go:356] "Generic (PLEG): container finished" podID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerID="78e8da45caf6cf507a1d7f396f8f9561b91ae98314aa0846e16ab7348550b3db" exitCode=60
Oct 31 00:30:02.489259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.488907    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerDied","Data":"78e8da45caf6cf507a1d7f396f8f9561b91ae98314aa0846e16ab7348550b3db"}
Oct 31 00:30:02.490536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.490283    5078 scope.go:122] "RemoveContainer" containerID="78e8da45caf6cf507a1d7f396f8f9561b91ae98314aa0846e16ab7348550b3db"
Oct 31 00:30:02.494277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.494252    5078 generic.go:356] "Generic (PLEG): container finished" podID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerID="0f8317a9acfc135ca42e279ecbb331c694597cde711f2683683de46e612da92b" exitCode=60
Oct 31 00:30:02.494613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.494578    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerDied","Data":"0f8317a9acfc135ca42e279ecbb331c694597cde711f2683683de46e612da92b"}
Oct 31 00:30:02.495816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.495800    5078 scope.go:122] "RemoveContainer" containerID="0f8317a9acfc135ca42e279ecbb331c694597cde711f2683683de46e612da92b"
Oct 31 00:30:02.505170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.505146    5078 generic.go:356] "Generic (PLEG): container finished" podID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerID="2e4b81664af35622a5065b52d3147c92b07330de48a50270924a7c549e243f66" exitCode=60
Oct 31 00:30:02.505421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.505401    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerDied","Data":"2e4b81664af35622a5065b52d3147c92b07330de48a50270924a7c549e243f66"}
Oct 31 00:30:02.506332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.506313    5078 scope.go:122] "RemoveContainer" containerID="2e4b81664af35622a5065b52d3147c92b07330de48a50270924a7c549e243f66"
Oct 31 00:30:02.510345 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.510317    5078 generic.go:356] "Generic (PLEG): container finished" podID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerID="382c900836188382f64d9b170bcf0e653b9d7fa51f49be1ff339c075c1725d74" exitCode=60
Oct 31 00:30:02.510447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.510366    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" event={"ID":"a04a4cd4-2941-4cd0-a378-7d590d70223f","Type":"ContainerDied","Data":"382c900836188382f64d9b170bcf0e653b9d7fa51f49be1ff339c075c1725d74"}
Oct 31 00:30:02.510822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:02.510799    5078 scope.go:122] "RemoveContainer" containerID="382c900836188382f64d9b170bcf0e653b9d7fa51f49be1ff339c075c1725d74"
Oct 31 00:30:03.549190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.549152    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerStarted","Data":"8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c"}
Oct 31 00:30:03.579555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.577761    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerStarted","Data":"b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf"}
Oct 31 00:30:03.582610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.582577    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerStarted","Data":"e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4"}
Oct 31 00:30:03.597046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.597014    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerStarted","Data":"d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062"}
Oct 31 00:30:03.605427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.605391    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" event={"ID":"a04a4cd4-2941-4cd0-a378-7d590d70223f","Type":"ContainerStarted","Data":"4001d40b950bb9ef3c5101e6d44fd2467e1378a07720f25bf937929d0ae2ed8b"}
Oct 31 00:30:03.618001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.617941    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" podStartSLOduration=3.617918783 podStartE2EDuration="3.617918783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.590777457 +0000 UTC m=+2171.537148175" watchObservedRunningTime="2026-10-31 00:30:03.617918783 +0000 UTC m=+2171.564289507"
Oct 31 00:30:03.705520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.705444    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" podStartSLOduration=3.705424637 podStartE2EDuration="3.705424637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.699376445 +0000 UTC m=+2171.645747161" watchObservedRunningTime="2026-10-31 00:30:03.705424637 +0000 UTC m=+2171.651795358"
Oct 31 00:30:03.720301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:03.720252    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" podStartSLOduration=3.720239036 podStartE2EDuration="3.720239036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.718160721 +0000 UTC m=+2171.664531438" watchObservedRunningTime="2026-10-31 00:30:03.720239036 +0000 UTC m=+2171.666609752"
Oct 31 00:30:04.319119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.319086    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"]
Oct 31 00:30:04.342586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.342072    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"]
Oct 31 00:30:04.621591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.621498    5078 generic.go:356] "Generic (PLEG): container finished" podID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerID="b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf" exitCode=60
Oct 31 00:30:04.622080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.621580    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerDied","Data":"b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf"}
Oct 31 00:30:04.622080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.621648    5078 scope.go:122] "RemoveContainer" containerID="78e8da45caf6cf507a1d7f396f8f9561b91ae98314aa0846e16ab7348550b3db"
Oct 31 00:30:04.622080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.621925    5078 scope.go:122] "RemoveContainer" containerID="b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf"
Oct 31 00:30:04.622330 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:04.622310    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7_odh-ai-gateway-infra(223c462f-c0fc-41ad-b18d-60b2dcb4f4ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca"
Oct 31 00:30:04.624359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.624337    5078 generic.go:356] "Generic (PLEG): container finished" podID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerID="e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4" exitCode=60
Oct 31 00:30:04.624452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.624419    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerDied","Data":"e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4"}
Oct 31 00:30:04.624692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.624678    5078 scope.go:122] "RemoveContainer" containerID="e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4"
Oct 31 00:30:04.625037 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:04.625015    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc_odh-ai-gateway-infra(12cc1b94-5558-4fc7-8c9d-7838af1454d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7"
Oct 31 00:30:04.626630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.626603    5078 generic.go:356] "Generic (PLEG): container finished" podID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerID="d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062" exitCode=60
Oct 31 00:30:04.626783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.626758    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerDied","Data":"d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062"}
Oct 31 00:30:04.626942 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.626926    5078 scope.go:122] "RemoveContainer" containerID="d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062"
Oct 31 00:30:04.627248 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:04.627226    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf_odh-ai-gateway-infra(bc181ef9-ba0b-41fc-9933-4e113f546ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8"
Oct 31 00:30:04.628975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.628954    5078 generic.go:356] "Generic (PLEG): container finished" podID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerID="4001d40b950bb9ef3c5101e6d44fd2467e1378a07720f25bf937929d0ae2ed8b" exitCode=60
Oct 31 00:30:04.629094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.629033    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" event={"ID":"a04a4cd4-2941-4cd0-a378-7d590d70223f","Type":"ContainerDied","Data":"4001d40b950bb9ef3c5101e6d44fd2467e1378a07720f25bf937929d0ae2ed8b"}
Oct 31 00:30:04.629282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.629265    5078 scope.go:122] "RemoveContainer" containerID="4001d40b950bb9ef3c5101e6d44fd2467e1378a07720f25bf937929d0ae2ed8b"
Oct 31 00:30:04.629611 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:04.629584    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt_odh-ai-gateway-infra(a04a4cd4-2941-4cd0-a378-7d590d70223f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f"
Oct 31 00:30:04.631236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.631214    5078 generic.go:356] "Generic (PLEG): container finished" podID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerID="8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c" exitCode=60
Oct 31 00:30:04.631346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.631307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerDied","Data":"8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c"}
Oct 31 00:30:04.631578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.631558    5078 scope.go:122] "RemoveContainer" containerID="8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c"
Oct 31 00:30:04.631823 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:04.631804    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj_odh-ai-gateway-infra(4bfeda2b-c240-4259-8ac0-0cffa34992de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de"
Oct 31 00:30:04.640400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.640383    5078 scope.go:122] "RemoveContainer" containerID="0f8317a9acfc135ca42e279ecbb331c694597cde711f2683683de46e612da92b"
Oct 31 00:30:04.660364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.660344    5078 scope.go:122] "RemoveContainer" containerID="2e4b81664af35622a5065b52d3147c92b07330de48a50270924a7c549e243f66"
Oct 31 00:30:04.677168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.677144    5078 scope.go:122] "RemoveContainer" containerID="382c900836188382f64d9b170bcf0e653b9d7fa51f49be1ff339c075c1725d74"
Oct 31 00:30:04.702226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.702203    5078 scope.go:122] "RemoveContainer" containerID="51445a9178e8369be2295eff4ac03dd6ad309c52756cd51dde99b682ae6f1de1"
Oct 31 00:30:04.828487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:04.827631    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"]
Oct 31 00:30:05.161637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.161603    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"]
Oct 31 00:30:05.161851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.161801    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2" podUID="c10674b7-2ea5-435a-8bdc-a0a706dc351a" containerName="maas-api" containerID="cri-o://6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1" gracePeriod=30
Oct 31 00:30:05.349939 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.349901    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"]
Oct 31 00:30:05.350302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.350189    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api" containerID="cri-o://3cd8f54c987c481753e4108b677a271ac5b2d85caa54a56a77a467f6e569ecc9" gracePeriod=30
Oct 31 00:30:05.398209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.398176    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"]
Oct 31 00:30:05.562703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.562677    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:30:05.751129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.751039    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm\" (UniqueName: \"kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm\") pod \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") "
Oct 31 00:30:05.758804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.758758    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm" pod "c10674b7-2ea5-435a-8bdc-a0a706dc351a" (UID: "c10674b7-2ea5-435a-8bdc-a0a706dc351a"). InnerVolumeSpecName "kube-api-access-wvqjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:05.802491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.793796    5078 generic.go:356] "Generic (PLEG): container finished" podID="c10674b7-2ea5-435a-8bdc-a0a706dc351a" containerID="6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1" exitCode=0
Oct 31 00:30:05.802491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.793917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2" event={"ID":"c10674b7-2ea5-435a-8bdc-a0a706dc351a","Type":"ContainerDied","Data":"6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1"}
Oct 31 00:30:05.802491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.793967    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2" event={"ID":"c10674b7-2ea5-435a-8bdc-a0a706dc351a","Type":"ContainerDied","Data":"532ccf4b977982f11cd87e99e33c1c3324f1db72ed9f10beddd9aff035841341"}
Oct 31 00:30:05.802491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.793995    5078 scope.go:122] "RemoveContainer" containerID="6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1"
Oct 31 00:30:05.802491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.794154    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"
Oct 31 00:30:05.823488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.821304    5078 scope.go:122] "RemoveContainer" containerID="d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062"
Oct 31 00:30:05.823488 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:05.821870    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf_odh-ai-gateway-infra(bc181ef9-ba0b-41fc-9933-4e113f546ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8"
Oct 31 00:30:05.827024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.826863    5078 scope.go:122] "RemoveContainer" containerID="4001d40b950bb9ef3c5101e6d44fd2467e1378a07720f25bf937929d0ae2ed8b"
Oct 31 00:30:05.832277 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:05.827252    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt_odh-ai-gateway-infra(a04a4cd4-2941-4cd0-a378-7d590d70223f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f"
Oct 31 00:30:05.835336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.834758    5078 scope.go:122] "RemoveContainer" containerID="6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1"
Oct 31 00:30:05.843854 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:05.843675    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1\": container with ID starting with 6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1 not found: ID does not exist" containerID="6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1"
Oct 31 00:30:05.843854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.843719    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1"} err="failed to get container status \"6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1\": rpc error: code = NotFound desc = could not find container \"6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1\": container with ID starting with 6010a2e526fdc44bc1eb154d10fbbaf1a9523270c4ad65efdeb5871099a1efb1 not found: ID does not exist"
Oct 31 00:30:05.846911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.846433    5078 generic.go:356] "Generic (PLEG): container finished" podID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerID="3cd8f54c987c481753e4108b677a271ac5b2d85caa54a56a77a467f6e569ecc9" exitCode=0
Oct 31 00:30:05.846911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.846624    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" event={"ID":"314a02c0-fdfd-4053-a213-5b8691190a5a","Type":"ContainerDied","Data":"3cd8f54c987c481753e4108b677a271ac5b2d85caa54a56a77a467f6e569ecc9"}
Oct 31 00:30:05.852962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.852932    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs\") pod \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") "
Oct 31 00:30:05.853085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.853020    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls\") pod \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") "
Oct 31 00:30:05.853317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.853299    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca\") pod \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\" (UID: \"c10674b7-2ea5-435a-8bdc-a0a706dc351a\") "
Oct 31 00:30:05.853907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.853885    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvqjm\" (UniqueName: \"kubernetes.io/projected/c10674b7-2ea5-435a-8bdc-a0a706dc351a-kube-api-access-wvqjm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:05.857119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.857091    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca" pod "c10674b7-2ea5-435a-8bdc-a0a706dc351a" (UID: "c10674b7-2ea5-435a-8bdc-a0a706dc351a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:05.858903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.858877    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls" pod "c10674b7-2ea5-435a-8bdc-a0a706dc351a" (UID: "c10674b7-2ea5-435a-8bdc-a0a706dc351a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:05.858998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.858913    5078 scope.go:122] "RemoveContainer" containerID="8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c"
Oct 31 00:30:05.859192 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:05.859168    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj_odh-ai-gateway-infra(4bfeda2b-c240-4259-8ac0-0cffa34992de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de"
Oct 31 00:30:05.859411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.859387    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs" pod "c10674b7-2ea5-435a-8bdc-a0a706dc351a" (UID: "c10674b7-2ea5-435a-8bdc-a0a706dc351a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:05.860091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.860053    5078 scope.go:122] "RemoveContainer" containerID="b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf"
Oct 31 00:30:05.860332 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:05.860289    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7_odh-ai-gateway-infra(223c462f-c0fc-41ad-b18d-60b2dcb4f4ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca"
Oct 31 00:30:05.863100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.863077    5078 scope.go:122] "RemoveContainer" containerID="e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4"
Oct 31 00:30:05.863381 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:05.863355    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc_odh-ai-gateway-infra(12cc1b94-5558-4fc7-8c9d-7838af1454d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7"
Oct 31 00:30:05.866035 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.866016    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:30:05.955264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.955220    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls\") pod \"314a02c0-fdfd-4053-a213-5b8691190a5a\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") "
Oct 31 00:30:05.955443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.955306    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs\") pod \"314a02c0-fdfd-4053-a213-5b8691190a5a\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") "
Oct 31 00:30:05.955443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.955396    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca\") pod \"314a02c0-fdfd-4053-a213-5b8691190a5a\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") "
Oct 31 00:30:05.955680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.955596    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz\" (UniqueName: \"kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz\") pod \"314a02c0-fdfd-4053-a213-5b8691190a5a\" (UID: \"314a02c0-fdfd-4053-a213-5b8691190a5a\") "
Oct 31 00:30:05.955847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.955810    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca" pod "314a02c0-fdfd-4053-a213-5b8691190a5a" (UID: "314a02c0-fdfd-4053-a213-5b8691190a5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:05.956380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.956354    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a02c0-fdfd-4053-a213-5b8691190a5a-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:05.956536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.956382    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10674b7-2ea5-435a-8bdc-a0a706dc351a-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:05.956536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.956399    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:05.956536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.956414    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c10674b7-2ea5-435a-8bdc-a0a706dc351a-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:05.957831 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.957809    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs" pod "314a02c0-fdfd-4053-a213-5b8691190a5a" (UID: "314a02c0-fdfd-4053-a213-5b8691190a5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:05.958329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.958302    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls" pod "314a02c0-fdfd-4053-a213-5b8691190a5a" (UID: "314a02c0-fdfd-4053-a213-5b8691190a5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:05.958428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:05.958409    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz" pod "314a02c0-fdfd-4053-a213-5b8691190a5a" (UID: "314a02c0-fdfd-4053-a213-5b8691190a5a"). InnerVolumeSpecName "kube-api-access-g7qbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:06.058040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.057945    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:06.058040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.057986    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314a02c0-fdfd-4053-a213-5b8691190a5a-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:06.058040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.058002    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7qbz\" (UniqueName: \"kubernetes.io/projected/314a02c0-fdfd-4053-a213-5b8691190a5a-kube-api-access-g7qbz\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:06.138494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.137331    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"]
Oct 31 00:30:06.143996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.143963    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d9875-64ddb96c8-2vpd2"]
Oct 31 00:30:06.198863 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.198821    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10674b7-2ea5-435a-8bdc-a0a706dc351a" path="/var/lib/kubelet/pods/c10674b7-2ea5-435a-8bdc-a0a706dc351a/volumes"
Oct 31 00:30:06.391336 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:06.391296    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:30:06.785766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.785677    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"]
Oct 31 00:30:06.786206 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.785942    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy" containerID="cri-o://15af77c0663c6c36392cd704dd7dec8a5dbc31d75b5973e50558bffa4c36350d" gracePeriod=30
Oct 31 00:30:06.877333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.876566    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"
Oct 31 00:30:06.877651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.877591    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824" event={"ID":"314a02c0-fdfd-4053-a213-5b8691190a5a","Type":"ContainerDied","Data":"a0608e0ec8ad7e1ec6d77be150f3c360970d8ed26e160b18d31a49de0e9be452"}
Oct 31 00:30:06.913039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:06.912535    5078 scope.go:122] "RemoveContainer" containerID="3cd8f54c987c481753e4108b677a271ac5b2d85caa54a56a77a467f6e569ecc9"
Oct 31 00:30:07.005212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.005147    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"]
Oct 31 00:30:07.046220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.046115    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4c360e-84f6dbf77f-zj824"]
Oct 31 00:30:07.175383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.175354    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:07.178234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.178211    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca\") pod \"a04a4cd4-2941-4cd0-a378-7d590d70223f\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") "
Oct 31 00:30:07.178408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.178391    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9\" (UniqueName: \"kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9\") pod \"a04a4cd4-2941-4cd0-a378-7d590d70223f\" (UID: \"a04a4cd4-2941-4cd0-a378-7d590d70223f\") "
Oct 31 00:30:07.178624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.178600    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca" pod "a04a4cd4-2941-4cd0-a378-7d590d70223f" (UID: "a04a4cd4-2941-4cd0-a378-7d590d70223f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:07.178965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.178948    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a4cd4-2941-4cd0-a378-7d590d70223f-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:07.180661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.180639    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9" pod "a04a4cd4-2941-4cd0-a378-7d590d70223f" (UID: "a04a4cd4-2941-4cd0-a378-7d590d70223f"). InnerVolumeSpecName "kube-api-access-x29z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:07.280850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.280814    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29z9\" (UniqueName: \"kubernetes.io/projected/a04a4cd4-2941-4cd0-a378-7d590d70223f-kube-api-access-x29z9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:07.507637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.507597    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"]
Oct 31 00:30:07.508786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.508765    5078 cpu_manager.go:414] "containerMap: removing container" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api"
Oct 31 00:30:07.508839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.508790    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api"
Oct 31 00:30:07.508839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.508810    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerName="cleanup"
Oct 31 00:30:07.508839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.508818    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerName="cleanup"
Oct 31 00:30:07.508948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.508838    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c10674b7-2ea5-435a-8bdc-a0a706dc351a" containerName="maas-api"
Oct 31 00:30:07.508948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.508846    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10674b7-2ea5-435a-8bdc-a0a706dc351a" containerName="maas-api"
Oct 31 00:30:07.509089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.509076    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" containerName="maas-api"
Oct 31 00:30:07.509122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.509097    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerName="cleanup"
Oct 31 00:30:07.509122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.509108    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10674b7-2ea5-435a-8bdc-a0a706dc351a" containerName="maas-api"
Oct 31 00:30:07.509184 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.509124    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerName="cleanup"
Oct 31 00:30:07.536960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.536635    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"]
Oct 31 00:30:07.536960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.536751    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.541567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.541543    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:07.584043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.584006    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.584228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.584072    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6lk\" (UniqueName: \"kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.584228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.584179    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.685902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.685865    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.685902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.685919    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6lk\" (UniqueName: \"kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.686182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.686020    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.686872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.686836    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.690244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.690214    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.701306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.701284    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6lk\" (UniqueName: \"kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk\") pod \"maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.855897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.855808    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:07.889618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.889582    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt" event={"ID":"a04a4cd4-2941-4cd0-a378-7d590d70223f","Type":"ContainerDied","Data":"427979a102cbf4d5092bdb4810fd1913cb48ed78dbbc2dd5713f2c462b402aa7"}
Oct 31 00:30:07.889802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.889655    5078 scope.go:122] "RemoveContainer" containerID="4001d40b950bb9ef3c5101e6d44fd2467e1378a07720f25bf937929d0ae2ed8b"
Oct 31 00:30:07.889872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.889810    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"
Oct 31 00:30:07.937473 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.937426    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"]
Oct 31 00:30:07.953710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:07.953674    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890110-rx7zt"]
Oct 31 00:30:08.119095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.119002    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"]
Oct 31 00:30:08.120155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.120131    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerName="cleanup"
Oct 31 00:30:08.120155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.120155    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" containerName="cleanup"
Oct 31 00:30:08.126304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.125433    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"]
Oct 31 00:30:08.126304 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.125575    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.130644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.130620    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.132085 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.132047    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"]
Oct 31 00:30:08.136592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.136566    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"]
Oct 31 00:30:08.170591 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:08.170566    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1218a7_d5bd_42e7_99d0_281a1090b882.slice/crio-c43006b16ca565b76208b6e646cd2b1addcd4e8eedb223b93fafca042d1cf69e WatchSource:0}: Error finding container c43006b16ca565b76208b6e646cd2b1addcd4e8eedb223b93fafca042d1cf69e: Status 404 returned error can't find the container with id c43006b16ca565b76208b6e646cd2b1addcd4e8eedb223b93fafca042d1cf69e
Oct 31 00:30:08.171803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.171768    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"]
Oct 31 00:30:08.203256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.202698    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.203256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.202845    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4vt\" (UniqueName: \"kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.203256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.202961    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.203256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.203097    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.203256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.203129    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.203256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.203155    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2l7z\" (UniqueName: \"kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.216677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.216645    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314a02c0-fdfd-4053-a213-5b8691190a5a" path="/var/lib/kubelet/pods/314a02c0-fdfd-4053-a213-5b8691190a5a/volumes"
Oct 31 00:30:08.217231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.217212    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04a4cd4-2941-4cd0-a378-7d590d70223f" path="/var/lib/kubelet/pods/a04a4cd4-2941-4cd0-a378-7d590d70223f/volumes"
Oct 31 00:30:08.309252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.309217    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.309252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.309261    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.309538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.309281    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2l7z\" (UniqueName: \"kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.309538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.309388    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.309538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.309452    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4vt\" (UniqueName: \"kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.309697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.309570    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.310143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.310118    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.310247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.310139    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.312198 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.312172    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.312669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.312647    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.322933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.322909    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4vt\" (UniqueName: \"kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt\") pod \"maas-api-revoke-sub-a9c5613e-4kmwf\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.323030 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.322999    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2l7z\" (UniqueName: \"kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z\") pod \"maas-api-revoke-sub-f4377892-66t8n\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.448188 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.448154    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:08.455768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.455734    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:08.756719 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:08.750518    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e27c39f_c611_489c_b51a_5fce76e4f52d.slice/crio-db8adcd9dd9dbb23cbbc68b8175e4023cc34db03418dd5a538e86c078e97c526 WatchSource:0}: Error finding container db8adcd9dd9dbb23cbbc68b8175e4023cc34db03418dd5a538e86c078e97c526: Status 404 returned error can't find the container with id db8adcd9dd9dbb23cbbc68b8175e4023cc34db03418dd5a538e86c078e97c526
Oct 31 00:30:08.756719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.751657    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"]
Oct 31 00:30:08.787689 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:08.787663    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d532e_e847_418a_a63e_990af75b81a7.slice/crio-d3b6026ca921f0d6cb8e762a4113d8485b9f3e096b71370bec91651d4dfe8493 WatchSource:0}: Error finding container d3b6026ca921f0d6cb8e762a4113d8485b9f3e096b71370bec91651d4dfe8493: Status 404 returned error can't find the container with id d3b6026ca921f0d6cb8e762a4113d8485b9f3e096b71370bec91651d4dfe8493
Oct 31 00:30:08.788066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.788042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"]
Oct 31 00:30:08.912253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.912216    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n" event={"ID":"4e27c39f-c611-489c-b51a-5fce76e4f52d","Type":"ContainerStarted","Data":"db8adcd9dd9dbb23cbbc68b8175e4023cc34db03418dd5a538e86c078e97c526"}
Oct 31 00:30:08.915405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.915374    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6" event={"ID":"0d1218a7-d5bd-42e7-99d0-281a1090b882","Type":"ContainerStarted","Data":"8ae0b072798b4abb138d89854dc3efad10f5ff9d5fa632c7108a5056aba84121"}
Oct 31 00:30:08.915559 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.915430    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6" event={"ID":"0d1218a7-d5bd-42e7-99d0-281a1090b882","Type":"ContainerStarted","Data":"c43006b16ca565b76208b6e646cd2b1addcd4e8eedb223b93fafca042d1cf69e"}
Oct 31 00:30:08.917152 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.917130    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf" event={"ID":"371d532e-e847-418a-a63e-990af75b81a7","Type":"ContainerStarted","Data":"d3b6026ca921f0d6cb8e762a4113d8485b9f3e096b71370bec91651d4dfe8493"}
Oct 31 00:30:08.942052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:08.941998    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6" podStartSLOduration=1.941985688 podStartE2EDuration="1.941985688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:08.938615434 +0000 UTC m=+2176.884986150" watchObservedRunningTime="2026-10-31 00:30:08.941985688 +0000 UTC m=+2176.888356405"
Oct 31 00:30:09.320172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.320119    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx" podUID="ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" containerName="payload-processing" containerID="cri-o://904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687" gracePeriod=25
Oct 31 00:30:09.342513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.342435    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv" podUID="97e0a354-0a16-4829-a1b5-b2528190b54e" containerName="payload-pre-processing" containerID="cri-o://1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0" gracePeriod=25
Oct 31 00:30:09.753524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.745888    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:30:09.802811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.802774    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:30:09.832314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.831623    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing" containerID="cri-o://42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231" gracePeriod=25
Oct 31 00:30:09.858376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.858257    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume\") pod \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") "
Oct 31 00:30:09.858686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.858653    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn\" (UniqueName: \"kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn\") pod \"97e0a354-0a16-4829-a1b5-b2528190b54e\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") "
Oct 31 00:30:09.859119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.859083    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume\") pod \"97e0a354-0a16-4829-a1b5-b2528190b54e\" (UID: \"97e0a354-0a16-4829-a1b5-b2528190b54e\") "
Oct 31 00:30:09.859356 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.859325    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4\" (UniqueName: \"kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4\") pod \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\" (UID: \"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3\") "
Oct 31 00:30:09.859679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.859633    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume" pod "97e0a354-0a16-4829-a1b5-b2528190b54e" (UID: "97e0a354-0a16-4829-a1b5-b2528190b54e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:09.860328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.860068    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume" pod "ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" (UID: "ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:09.860328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.860275    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e0a354-0a16-4829-a1b5-b2528190b54e-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:09.860328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.860296    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:09.863729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.863686    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4" pod "ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" (UID: "ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3"). InnerVolumeSpecName "kube-api-access-br6z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:09.874500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.874054    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn" pod "97e0a354-0a16-4829-a1b5-b2528190b54e" (UID: "97e0a354-0a16-4829-a1b5-b2528190b54e"). InnerVolumeSpecName "kube-api-access-dlbnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:09.935326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.935285    5078 generic.go:356] "Generic (PLEG): container finished" podID="371d532e-e847-418a-a63e-990af75b81a7" containerID="86bb60ab180293e9d2dd94855ef954e1f09ab8fa80b45e5eeaace039a57bc486" exitCode=0
Oct 31 00:30:09.935783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.935405    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf" event={"ID":"371d532e-e847-418a-a63e-990af75b81a7","Type":"ContainerDied","Data":"86bb60ab180293e9d2dd94855ef954e1f09ab8fa80b45e5eeaace039a57bc486"}
Oct 31 00:30:09.937740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.937542    5078 generic.go:356] "Generic (PLEG): container finished" podID="ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" containerID="904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687" exitCode=0
Oct 31 00:30:09.937740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.937599    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"
Oct 31 00:30:09.937740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.937631    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx" event={"ID":"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3","Type":"ContainerDied","Data":"904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687"}
Oct 31 00:30:09.937740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.937690    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx" event={"ID":"ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3","Type":"ContainerDied","Data":"f67e428899b331b0044fb3142ac92cec91caa829fc0b78160ae66000fceb664f"}
Oct 31 00:30:09.937740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.937719    5078 scope.go:122] "RemoveContainer" containerID="904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687"
Oct 31 00:30:09.940643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.940620    5078 generic.go:356] "Generic (PLEG): container finished" podID="4e27c39f-c611-489c-b51a-5fce76e4f52d" containerID="a8424a436e1a69ee2bb7bdd461d6a6f863a3b054b5c903a7d39fcc66fea3c965" exitCode=0
Oct 31 00:30:09.940786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.940712    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n" event={"ID":"4e27c39f-c611-489c-b51a-5fce76e4f52d","Type":"ContainerDied","Data":"a8424a436e1a69ee2bb7bdd461d6a6f863a3b054b5c903a7d39fcc66fea3c965"}
Oct 31 00:30:09.942615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.942595    5078 generic.go:356] "Generic (PLEG): container finished" podID="0d1218a7-d5bd-42e7-99d0-281a1090b882" containerID="8ae0b072798b4abb138d89854dc3efad10f5ff9d5fa632c7108a5056aba84121" exitCode=0
Oct 31 00:30:09.942719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.942677    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6" event={"ID":"0d1218a7-d5bd-42e7-99d0-281a1090b882","Type":"ContainerDied","Data":"8ae0b072798b4abb138d89854dc3efad10f5ff9d5fa632c7108a5056aba84121"}
Oct 31 00:30:09.944416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.944396    5078 generic.go:356] "Generic (PLEG): container finished" podID="97e0a354-0a16-4829-a1b5-b2528190b54e" containerID="1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0" exitCode=0
Oct 31 00:30:09.944550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.944433    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"
Oct 31 00:30:09.944550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.944504    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv" event={"ID":"97e0a354-0a16-4829-a1b5-b2528190b54e","Type":"ContainerDied","Data":"1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0"}
Oct 31 00:30:09.944550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.944543    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv" event={"ID":"97e0a354-0a16-4829-a1b5-b2528190b54e","Type":"ContainerDied","Data":"96dac526a761e9981394110ac6822624cf360dc24e06e18e397124b90ff93a82"}
Oct 31 00:30:09.961799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.961767    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br6z4\" (UniqueName: \"kubernetes.io/projected/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3-kube-api-access-br6z4\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:09.961799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:09.961798    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbnn\" (UniqueName: \"kubernetes.io/projected/97e0a354-0a16-4829-a1b5-b2528190b54e-kube-api-access-dlbnn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:10.052207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.051938    5078 scope.go:122] "RemoveContainer" containerID="904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687"
Oct 31 00:30:10.053739 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:10.053575    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687\": container with ID starting with 904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687 not found: ID does not exist" containerID="904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687"
Oct 31 00:30:10.053739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.053618    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687"} err="failed to get container status \"904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687\": rpc error: code = NotFound desc = could not find container \"904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687\": container with ID starting with 904ac03d3c043aed5f7a6896313a8322216613b0184ea4921ed918139f2dc687 not found: ID does not exist"
Oct 31 00:30:10.053739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.053641    5078 scope.go:122] "RemoveContainer" containerID="1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0"
Oct 31 00:30:10.068574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.068539    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"]
Oct 31 00:30:10.073514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.073483    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d4ad9-96ddb5fbc-gdwcv"]
Oct 31 00:30:10.076592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.076425    5078 scope.go:122] "RemoveContainer" containerID="1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0"
Oct 31 00:30:10.077186 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:10.077158    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0\": container with ID starting with 1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0 not found: ID does not exist" containerID="1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0"
Oct 31 00:30:10.077350 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.077326    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0"} err="failed to get container status \"1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0\": rpc error: code = NotFound desc = could not find container \"1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0\": container with ID starting with 1e8faae9a48543eb3b0c6b56073a39e6798e4e29635e99c055c91f20463325b0 not found: ID does not exist"
Oct 31 00:30:10.109648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.106391    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"]
Oct 31 00:30:10.112742 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.112709    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d4ad9-6774966877-8tjgx"]
Oct 31 00:30:10.200364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.200322    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e0a354-0a16-4829-a1b5-b2528190b54e" path="/var/lib/kubelet/pods/97e0a354-0a16-4829-a1b5-b2528190b54e/volumes"
Oct 31 00:30:10.201492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:10.201441    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" path="/var/lib/kubelet/pods/ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3/volumes"
Oct 31 00:30:11.430602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.389271    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:11.619672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.615109    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca\") pod \"4e27c39f-c611-489c-b51a-5fce76e4f52d\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") "
Oct 31 00:30:11.619672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.615241    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z\" (UniqueName: \"kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z\") pod \"4e27c39f-c611-489c-b51a-5fce76e4f52d\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") "
Oct 31 00:30:11.619672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.615291    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca\") pod \"4e27c39f-c611-489c-b51a-5fce76e4f52d\" (UID: \"4e27c39f-c611-489c-b51a-5fce76e4f52d\") "
Oct 31 00:30:11.668491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.637813    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca" pod "4e27c39f-c611-489c-b51a-5fce76e4f52d" (UID: "4e27c39f-c611-489c-b51a-5fce76e4f52d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:11.726574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.725437    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca" pod "4e27c39f-c611-489c-b51a-5fce76e4f52d" (UID: "4e27c39f-c611-489c-b51a-5fce76e4f52d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:11.743761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.740977    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.743761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.741012    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e27c39f-c611-489c-b51a-5fce76e4f52d-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.743761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.741729    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z" pod "4e27c39f-c611-489c-b51a-5fce76e4f52d" (UID: "4e27c39f-c611-489c-b51a-5fce76e4f52d"). InnerVolumeSpecName "kube-api-access-n2l7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:11.744731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.744216    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:11.768006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.757579    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:11.855489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.854932    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca\") pod \"371d532e-e847-418a-a63e-990af75b81a7\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") "
Oct 31 00:30:11.855489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.854990    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca\") pod \"0d1218a7-d5bd-42e7-99d0-281a1090b882\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") "
Oct 31 00:30:11.855489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.855114    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk\" (UniqueName: \"kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk\") pod \"0d1218a7-d5bd-42e7-99d0-281a1090b882\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") "
Oct 31 00:30:11.862234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.860956    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk" pod "0d1218a7-d5bd-42e7-99d0-281a1090b882" (UID: "0d1218a7-d5bd-42e7-99d0-281a1090b882"). InnerVolumeSpecName "kube-api-access-wg6lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:11.867873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.866181    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt\" (UniqueName: \"kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt\") pod \"371d532e-e847-418a-a63e-990af75b81a7\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") "
Oct 31 00:30:11.867873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.866293    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca\") pod \"0d1218a7-d5bd-42e7-99d0-281a1090b882\" (UID: \"0d1218a7-d5bd-42e7-99d0-281a1090b882\") "
Oct 31 00:30:11.867873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.866364    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca\") pod \"371d532e-e847-418a-a63e-990af75b81a7\" (UID: \"371d532e-e847-418a-a63e-990af75b81a7\") "
Oct 31 00:30:11.867873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.866806    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca" pod "0d1218a7-d5bd-42e7-99d0-281a1090b882" (UID: "0d1218a7-d5bd-42e7-99d0-281a1090b882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:11.867873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.867409    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca" pod "371d532e-e847-418a-a63e-990af75b81a7" (UID: "371d532e-e847-418a-a63e-990af75b81a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:11.869539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.869382    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt" pod "371d532e-e847-418a-a63e-990af75b81a7" (UID: "371d532e-e847-418a-a63e-990af75b81a7"). InnerVolumeSpecName "kube-api-access-gr4vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:11.870577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.869995    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d532e-e847-418a-a63e-990af75b81a7-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.870577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.870025    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6lk\" (UniqueName: \"kubernetes.io/projected/0d1218a7-d5bd-42e7-99d0-281a1090b882-kube-api-access-wg6lk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.870577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.870040    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2l7z\" (UniqueName: \"kubernetes.io/projected/4e27c39f-c611-489c-b51a-5fce76e4f52d-kube-api-access-n2l7z\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.870577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.870065    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.880504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.872888    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca" pod "371d532e-e847-418a-a63e-990af75b81a7" (UID: "371d532e-e847-418a-a63e-990af75b81a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:11.880504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.874983    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca" pod "0d1218a7-d5bd-42e7-99d0-281a1090b882" (UID: "0d1218a7-d5bd-42e7-99d0-281a1090b882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:11.971709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.971672    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d532e-e847-418a-a63e-990af75b81a7-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.971709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.971708    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1218a7-d5bd-42e7-99d0-281a1090b882-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.971927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.971723    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4vt\" (UniqueName: \"kubernetes.io/projected/371d532e-e847-418a-a63e-990af75b81a7-kube-api-access-gr4vt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:11.992222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.992187    5078 generic.go:356] "Generic (PLEG): container finished" podID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerID="15af77c0663c6c36392cd704dd7dec8a5dbc31d75b5973e50558bffa4c36350d" exitCode=0
Oct 31 00:30:11.992380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.992259    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" event={"ID":"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da","Type":"ContainerDied","Data":"15af77c0663c6c36392cd704dd7dec8a5dbc31d75b5973e50558bffa4c36350d"}
Oct 31 00:30:11.992380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.992325    5078 scope.go:122] "RemoveContainer" containerID="1a630d9284e37575cd1158c5a5edef094977a44e1abe5ed667cc27f72e85717a"
Oct 31 00:30:11.995771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.995740    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf" event={"ID":"371d532e-e847-418a-a63e-990af75b81a7","Type":"ContainerDied","Data":"d3b6026ca921f0d6cb8e762a4113d8485b9f3e096b71370bec91651d4dfe8493"}
Oct 31 00:30:11.995882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.995796    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b6026ca921f0d6cb8e762a4113d8485b9f3e096b71370bec91651d4dfe8493"
Oct 31 00:30:11.995882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:11.995862    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"
Oct 31 00:30:12.019487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.016236    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n" event={"ID":"4e27c39f-c611-489c-b51a-5fce76e4f52d","Type":"ContainerDied","Data":"db8adcd9dd9dbb23cbbc68b8175e4023cc34db03418dd5a538e86c078e97c526"}
Oct 31 00:30:12.019487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.016301    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db8adcd9dd9dbb23cbbc68b8175e4023cc34db03418dd5a538e86c078e97c526"
Oct 31 00:30:12.019487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.016382    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"
Oct 31 00:30:12.034029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.031441    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6" event={"ID":"0d1218a7-d5bd-42e7-99d0-281a1090b882","Type":"ContainerDied","Data":"c43006b16ca565b76208b6e646cd2b1addcd4e8eedb223b93fafca042d1cf69e"}
Oct 31 00:30:12.034029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.031524    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43006b16ca565b76208b6e646cd2b1addcd4e8eedb223b93fafca042d1cf69e"
Oct 31 00:30:12.034029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.031596    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"
Oct 31 00:30:12.215290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.215259    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:30:12.299116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299078    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.299432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299401    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:12.299634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299607    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.299758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299660    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.299826 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299755    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.299891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299870    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:12.299957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299894    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.299957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.299930    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.300269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.300045    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.300269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.300073    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.300269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.300148    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs\") pod \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\" (UID: \"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da\") "
Oct 31 00:30:12.300437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.300402    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:12.300909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.300884    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.301041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.301026    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.301137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.301124    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.301445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.301424    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:12.303139 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.302922    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:30:12.303350 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.303282    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:12.306540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.306451    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:12.306642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.306573    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "kube-api-access-7kzpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:12.308943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.308919    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy" pod "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" (UID: "f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:12.402927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.402896    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.402927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.402924    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzpc\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-kube-api-access-7kzpc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.402927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.402936    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.403180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.402945    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.403180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.402955    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.403180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.402963    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:12.826992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:12.826884    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:13.042225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:13.042195    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"
Oct 31 00:30:13.042402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:13.042235    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76" event={"ID":"f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da","Type":"ContainerDied","Data":"079a0810048bcf0fb8174028536b4699460b6e5d166b18928e4c03735d565c74"}
Oct 31 00:30:13.042402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:13.042300    5078 scope.go:122] "RemoveContainer" containerID="15af77c0663c6c36392cd704dd7dec8a5dbc31d75b5973e50558bffa4c36350d"
Oct 31 00:30:13.077112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:13.077034    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"]
Oct 31 00:30:13.086222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:13.086195    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d898f3-openshift-default-6cc9655-xlf76"]
Oct 31 00:30:14.201146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.201110    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" path="/var/lib/kubelet/pods/f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da/volumes"
Oct 31 00:30:14.399305 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.399257    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main" containerID="cri-o://7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469" gracePeriod=45
Oct 31 00:30:14.411986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.411945    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" containerID="cri-o://47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42" gracePeriod=45
Oct 31 00:30:14.445865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.445828    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 00:30:14.767347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.767277    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:30:14.776401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.776374    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:30:14.791650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.791621    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"]
Oct 31 00:30:14.855618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855586    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d\" (UniqueName: \"kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d\") pod \"f5c46e27-4680-4481-9471-a06b9965262a\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") "
Oct 31 00:30:14.855801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855722    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache\") pod \"f5c46e27-4680-4481-9471-a06b9965262a\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") "
Oct 31 00:30:14.855871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855801    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs\" (UniqueName: \"kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs\") pod \"66848bbb-e0ec-48b2-9154-55eec47a4122\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") "
Oct 31 00:30:14.855871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855839    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home\") pod \"66848bbb-e0ec-48b2-9154-55eec47a4122\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") "
Oct 31 00:30:14.855990 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855886    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache\") pod \"66848bbb-e0ec-48b2-9154-55eec47a4122\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") "
Oct 31 00:30:14.855990 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855912    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home\") pod \"f5c46e27-4680-4481-9471-a06b9965262a\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") "
Oct 31 00:30:14.856592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.855974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir\") pod \"f5c46e27-4680-4481-9471-a06b9965262a\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") "
Oct 31 00:30:14.856708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856274    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home" pod "66848bbb-e0ec-48b2-9154-55eec47a4122" (UID: "66848bbb-e0ec-48b2-9154-55eec47a4122"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.856708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856316    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache" pod "f5c46e27-4680-4481-9471-a06b9965262a" (UID: "f5c46e27-4680-4481-9471-a06b9965262a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.856708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856505    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache" pod "66848bbb-e0ec-48b2-9154-55eec47a4122" (UID: "66848bbb-e0ec-48b2-9154-55eec47a4122"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.856708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856521    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir" pod "f5c46e27-4680-4481-9471-a06b9965262a" (UID: "f5c46e27-4680-4481-9471-a06b9965262a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.856708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856555    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home" pod "f5c46e27-4680-4481-9471-a06b9965262a" (UID: "f5c46e27-4680-4481-9471-a06b9965262a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.856969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856715    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir\") pod \"66848bbb-e0ec-48b2-9154-55eec47a4122\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") "
Oct 31 00:30:14.856969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856946    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir" pod "66848bbb-e0ec-48b2-9154-55eec47a4122" (UID: "66848bbb-e0ec-48b2-9154-55eec47a4122"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.857243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.856746    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm\") pod \"66848bbb-e0ec-48b2-9154-55eec47a4122\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") "
Oct 31 00:30:14.857381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.857290    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm\") pod \"f5c46e27-4680-4481-9471-a06b9965262a\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") "
Oct 31 00:30:14.857853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.857640    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs\") pod \"f5c46e27-4680-4481-9471-a06b9965262a\" (UID: \"f5c46e27-4680-4481-9471-a06b9965262a\") "
Oct 31 00:30:14.858233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.858193    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc\" (UniqueName: \"kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc\") pod \"66848bbb-e0ec-48b2-9154-55eec47a4122\" (UID: \"66848bbb-e0ec-48b2-9154-55eec47a4122\") "
Oct 31 00:30:14.859359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.859334    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.859449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.859364    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.859449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.859378    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.859449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.859391    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.859449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.859404    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.859449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.859416    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.861949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.861924    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm" pod "66848bbb-e0ec-48b2-9154-55eec47a4122" (UID: "66848bbb-e0ec-48b2-9154-55eec47a4122"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.862056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.861982    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm" pod "f5c46e27-4680-4481-9471-a06b9965262a" (UID: "f5c46e27-4680-4481-9471-a06b9965262a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:14.862149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.862129    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs" pod "66848bbb-e0ec-48b2-9154-55eec47a4122" (UID: "66848bbb-e0ec-48b2-9154-55eec47a4122"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:14.862267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.862250    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc" pod "66848bbb-e0ec-48b2-9154-55eec47a4122" (UID: "66848bbb-e0ec-48b2-9154-55eec47a4122"). InnerVolumeSpecName "kube-api-access-s5xxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:14.863795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.863769    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs" pod "f5c46e27-4680-4481-9471-a06b9965262a" (UID: "f5c46e27-4680-4481-9471-a06b9965262a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:14.865054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.865029    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d" pod "f5c46e27-4680-4481-9471-a06b9965262a" (UID: "f5c46e27-4680-4481-9471-a06b9965262a"). InnerVolumeSpecName "kube-api-access-qd86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:14.961150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.961110    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66848bbb-e0ec-48b2-9154-55eec47a4122-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.961150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.961142    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5c46e27-4680-4481-9471-a06b9965262a-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.961150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.961155    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5c46e27-4680-4481-9471-a06b9965262a-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.961407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.961169    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xxc\" (UniqueName: \"kubernetes.io/projected/66848bbb-e0ec-48b2-9154-55eec47a4122-kube-api-access-s5xxc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.961407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.961183    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd86d\" (UniqueName: \"kubernetes.io/projected/f5c46e27-4680-4481-9471-a06b9965262a-kube-api-access-qd86d\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:14.961407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:14.961195    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66848bbb-e0ec-48b2-9154-55eec47a4122-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:15.114294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.114190    5078 generic.go:356] "Generic (PLEG): container finished" podID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerID="47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42" exitCode=0
Oct 31 00:30:15.114562 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.114544    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"
Oct 31 00:30:15.115528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.115497    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" event={"ID":"66848bbb-e0ec-48b2-9154-55eec47a4122","Type":"ContainerDied","Data":"47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42"}
Oct 31 00:30:15.115664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.115548    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk" event={"ID":"66848bbb-e0ec-48b2-9154-55eec47a4122","Type":"ContainerDied","Data":"c2805bf59dd98e564946b21ce1c52bdbbefbe8497bff148f487a2a638e4964d5"}
Oct 31 00:30:15.115664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.115566    5078 scope.go:122] "RemoveContainer" containerID="47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42"
Oct 31 00:30:15.116877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.116729    5078 generic.go:356] "Generic (PLEG): container finished" podID="f5c46e27-4680-4481-9471-a06b9965262a" containerID="7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469" exitCode=0
Oct 31 00:30:15.116877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.116836    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" event={"ID":"f5c46e27-4680-4481-9471-a06b9965262a","Type":"ContainerDied","Data":"7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469"}
Oct 31 00:30:15.116877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.116858    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d" event={"ID":"f5c46e27-4680-4481-9471-a06b9965262a","Type":"ContainerDied","Data":"f9eb1dc9d62a419d25986ff042c7aec28cde06cb2f7617d59a233c9e523d48a1"}
Oct 31 00:30:15.117049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.116909    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"
Oct 31 00:30:15.128553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.128534    5078 scope.go:122] "RemoveContainer" containerID="47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42"
Oct 31 00:30:15.129100 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:15.129080    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42\": container with ID starting with 47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42 not found: ID does not exist" containerID="47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42"
Oct 31 00:30:15.129185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.129107    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42"} err="failed to get container status \"47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42\": rpc error: code = NotFound desc = could not find container \"47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42\": container with ID starting with 47efac742153f121211e4bb291b9d8d9f19c1484274f475d1b28c0c1dad37c42 not found: ID does not exist"
Oct 31 00:30:15.129185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.129122    5078 scope.go:122] "RemoveContainer" containerID="7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469"
Oct 31 00:30:15.140185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.140162    5078 scope.go:122] "RemoveContainer" containerID="7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469"
Oct 31 00:30:15.140497 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:15.140450    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469\": container with ID starting with 7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469 not found: ID does not exist" containerID="7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469"
Oct 31 00:30:15.140567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.140494    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469"} err="failed to get container status \"7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469\": rpc error: code = NotFound desc = could not find container \"7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469\": container with ID starting with 7be27633c8d8912d4e2ec04871a41ec14bde4de9ee4885a1ea8cde118f210469 not found: ID does not exist"
Oct 31 00:30:15.151264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.151239    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"]
Oct 31 00:30:15.160937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.160902    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/simulator-w4-d91de6-kserve-5656898d8-pdcdk"]
Oct 31 00:30:15.186334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.186302    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"]
Oct 31 00:30:15.187348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.187316    5078 scope.go:122] "RemoveContainer" containerID="b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf"
Oct 31 00:30:15.194525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:15.191689    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/premium-w4-d91de6-kserve-6f4b8f47f7-hpm6d"]
Oct 31 00:30:16.133484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:16.133427    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerStarted","Data":"60fb0a09f7fbadb2041542237e4ec35f38876c0af15dbd5539913ed1368b749b"}
Oct 31 00:30:16.188032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:16.187868    5078 scope.go:122] "RemoveContainer" containerID="8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c"
Oct 31 00:30:16.199297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:16.199264    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" path="/var/lib/kubelet/pods/66848bbb-e0ec-48b2-9154-55eec47a4122/volumes"
Oct 31 00:30:16.199913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:16.199892    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c46e27-4680-4481-9471-a06b9965262a" path="/var/lib/kubelet/pods/f5c46e27-4680-4481-9471-a06b9965262a/volumes"
Oct 31 00:30:16.693927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:16.693895    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"]
Oct 31 00:30:16.694177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:16.694152    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy" containerID="cri-o://a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7" gracePeriod=30
Oct 31 00:30:17.146856 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.146818    5078 generic.go:356] "Generic (PLEG): container finished" podID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerID="60fb0a09f7fbadb2041542237e4ec35f38876c0af15dbd5539913ed1368b749b" exitCode=60
Oct 31 00:30:17.147308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.146894    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerDied","Data":"60fb0a09f7fbadb2041542237e4ec35f38876c0af15dbd5539913ed1368b749b"}
Oct 31 00:30:17.147308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.146970    5078 scope.go:122] "RemoveContainer" containerID="b51f56e6c77caddeb78b75b213cf01b90a9da251917a306bb6b4c39dcb1476cf"
Oct 31 00:30:17.147536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.147514    5078 scope.go:122] "RemoveContainer" containerID="60fb0a09f7fbadb2041542237e4ec35f38876c0af15dbd5539913ed1368b749b"
Oct 31 00:30:17.147955 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:17.147925    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7_odh-ai-gateway-infra(223c462f-c0fc-41ad-b18d-60b2dcb4f4ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca"
Oct 31 00:30:17.149817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.149793    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerStarted","Data":"4f4bff6f7873fe137e291901e9446c342ba2b43837d6b09dfc4b4a7793fbed1a"}
Oct 31 00:30:17.180227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.180197    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"]
Oct 31 00:30:17.187847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.187814    5078 scope.go:122] "RemoveContainer" containerID="d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062"
Oct 31 00:30:17.396405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.396374    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"]
Oct 31 00:30:17.397175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397129    5078 cpu_manager.go:414] "containerMap: removing container" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main"
Oct 31 00:30:17.397175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397150    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397178    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4e27c39f-c611-489c-b51a-5fce76e4f52d" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397184    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e27c39f-c611-489c-b51a-5fce76e4f52d" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397198    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397203    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397211    5078 cpu_manager.go:414] "containerMap: removing container" podUID="97e0a354-0a16-4829-a1b5-b2528190b54e" containerName="payload-pre-processing"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397216    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e0a354-0a16-4829-a1b5-b2528190b54e" containerName="payload-pre-processing"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397229    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397234    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397248    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" containerName="payload-processing"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397252    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" containerName="payload-processing"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397262    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0d1218a7-d5bd-42e7-99d0-281a1090b882" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397267    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1218a7-d5bd-42e7-99d0-281a1090b882" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397275    5078 cpu_manager.go:414] "containerMap: removing container" podUID="371d532e-e847-418a-a63e-990af75b81a7" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397279    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371d532e-e847-418a-a63e-990af75b81a7" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397412    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2b3c00-cdd3-489a-9c9f-4fdfa1df79d3" containerName="payload-processing"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397425    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1218a7-d5bd-42e7-99d0-281a1090b882" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397433    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="371d532e-e847-418a-a63e-990af75b81a7" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397439    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397452    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="66848bbb-e0ec-48b2-9154-55eec47a4122" containerName="main"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397471    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e27c39f-c611-489c-b51a-5fce76e4f52d" containerName="revoke-keys"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397478    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c46e27-4680-4481-9471-a06b9965262a" containerName="main"
Oct 31 00:30:17.397579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.397484    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e0a354-0a16-4829-a1b5-b2528190b54e" containerName="payload-pre-processing"
Oct 31 00:30:17.402238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.402219    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.420394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.420342    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:17.423898 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.423878    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"]
Oct 31 00:30:17.510240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.509428    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.519024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.518988    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvvf\" (UniqueName: \"kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.519578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.519551    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.621693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.621658    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.621889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.621786    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvvf\" (UniqueName: \"kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.621889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.621884    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.622567 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.622538    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.624177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.624150    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.638283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.638253    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvvf\" (UniqueName: \"kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf\") pod \"maas-api-revoke-keys-e2e-api-b-31108d-5m4pq\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:17.660202 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.660137    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"]
Oct 31 00:30:17.701595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.701563    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"]
Oct 31 00:30:17.721223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:17.721189    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:18.040996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.040916    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"]
Oct 31 00:30:18.041219 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.041177    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" containerID="cri-o://faba5592806f102eccc49d8f9f1b7675d239e628fa99db84b139e966000b332b" gracePeriod=30
Oct 31 00:30:18.049642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.049614    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"]
Oct 31 00:30:18.050956 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:18.050935    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ad2eba_5caa_4231_87d5_50431b292d29.slice/crio-1d7852e4b99eb12ec91ac061617dafb45efe3a101166c314850e2fb6a5321855 WatchSource:0}: Error finding container 1d7852e4b99eb12ec91ac061617dafb45efe3a101166c314850e2fb6a5321855: Status 404 returned error can't find the container with id 1d7852e4b99eb12ec91ac061617dafb45efe3a101166c314850e2fb6a5321855
Oct 31 00:30:18.164335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.164299    5078 generic.go:356] "Generic (PLEG): container finished" podID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerID="4f4bff6f7873fe137e291901e9446c342ba2b43837d6b09dfc4b4a7793fbed1a" exitCode=60
Oct 31 00:30:18.164947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.164401    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerDied","Data":"4f4bff6f7873fe137e291901e9446c342ba2b43837d6b09dfc4b4a7793fbed1a"}
Oct 31 00:30:18.164947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.164555    5078 scope.go:122] "RemoveContainer" containerID="8031c4c7e05f180486f554063db94b4132c125c1435d5344d4759910f03ea07c"
Oct 31 00:30:18.165120 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.164990    5078 scope.go:122] "RemoveContainer" containerID="4f4bff6f7873fe137e291901e9446c342ba2b43837d6b09dfc4b4a7793fbed1a"
Oct 31 00:30:18.165454 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:18.165425    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj_odh-ai-gateway-infra(4bfeda2b-c240-4259-8ac0-0cffa34992de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de"
Oct 31 00:30:18.172080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.172054    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq" event={"ID":"f4ad2eba-5caa-4231-87d5-50431b292d29","Type":"ContainerStarted","Data":"1d7852e4b99eb12ec91ac061617dafb45efe3a101166c314850e2fb6a5321855"}
Oct 31 00:30:18.175628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.175607    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerStarted","Data":"b087c2d8d321a2e2952003da621cccd2e09d5b82d1b2f589d6a67a6eba3c5fdb"}
Oct 31 00:30:18.277621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.277207    5078 scope.go:122] "RemoveContainer" containerID="e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4"
Oct 31 00:30:18.328549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.325729    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"]
Oct 31 00:30:18.520653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.520628    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:18.536657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.536631    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca\") pod \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") "
Oct 31 00:30:18.536822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.536807    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp\" (UniqueName: \"kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp\") pod \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\" (UID: \"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca\") "
Oct 31 00:30:18.536956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.536934    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca" pod "223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" (UID: "223c462f-c0fc-41ad-b18d-60b2dcb4f4ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:18.537366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.537345    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:18.570265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.570181    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp" pod "223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" (UID: "223c462f-c0fc-41ad-b18d-60b2dcb4f4ca"). InnerVolumeSpecName "kube-api-access-nbqlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:18.638841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:18.638808    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbqlp\" (UniqueName: \"kubernetes.io/projected/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca-kube-api-access-nbqlp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.188301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.188266    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerStarted","Data":"1c4d88dd9472d4bffd5a6d1bb9205111ae45b82cce4d0c6b102b24826f17d5e2"}
Oct 31 00:30:19.190806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.190781    5078 generic.go:356] "Generic (PLEG): container finished" podID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerID="b087c2d8d321a2e2952003da621cccd2e09d5b82d1b2f589d6a67a6eba3c5fdb" exitCode=60
Oct 31 00:30:19.190962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.190947    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerDied","Data":"b087c2d8d321a2e2952003da621cccd2e09d5b82d1b2f589d6a67a6eba3c5fdb"}
Oct 31 00:30:19.191028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.190985    5078 scope.go:122] "RemoveContainer" containerID="d39948d1ced8c1113c18eb064f2ada4eff96f06c85cba08fac991499f40dc062"
Oct 31 00:30:19.191327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.191311    5078 scope.go:122] "RemoveContainer" containerID="b087c2d8d321a2e2952003da621cccd2e09d5b82d1b2f589d6a67a6eba3c5fdb"
Oct 31 00:30:19.191779 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:19.191679    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf_odh-ai-gateway-infra(bc181ef9-ba0b-41fc-9933-4e113f546ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8"
Oct 31 00:30:19.195048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.195019    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"
Oct 31 00:30:19.195447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.195016    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7" event={"ID":"223c462f-c0fc-41ad-b18d-60b2dcb4f4ca","Type":"ContainerDied","Data":"974b85ba056f0f156f277a38785a415fef07199f6d59ab7d24afb1b230fe4b68"}
Oct 31 00:30:19.200691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.200663    5078 generic.go:356] "Generic (PLEG): container finished" podID="f4ad2eba-5caa-4231-87d5-50431b292d29" containerID="31df9b0a2cf1fbd49840ae0965e048620809e509b1580608cfbaa372852e01be" exitCode=0
Oct 31 00:30:19.200791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.200701    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq" event={"ID":"f4ad2eba-5caa-4231-87d5-50431b292d29","Type":"ContainerDied","Data":"31df9b0a2cf1fbd49840ae0965e048620809e509b1580608cfbaa372852e01be"}
Oct 31 00:30:19.258423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.258115    5078 scope.go:122] "RemoveContainer" containerID="60fb0a09f7fbadb2041542237e4ec35f38876c0af15dbd5539913ed1368b749b"
Oct 31 00:30:19.292402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.292361    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"]
Oct 31 00:30:19.420887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.420813    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"]
Oct 31 00:30:19.432449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.432424    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1c9660-29890110-57kj7"]
Oct 31 00:30:19.472428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.472393    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:19.554576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.554544    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr\" (UniqueName: \"kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr\") pod \"4bfeda2b-c240-4259-8ac0-0cffa34992de\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") "
Oct 31 00:30:19.554761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.554647    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca\") pod \"4bfeda2b-c240-4259-8ac0-0cffa34992de\" (UID: \"4bfeda2b-c240-4259-8ac0-0cffa34992de\") "
Oct 31 00:30:19.555038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.555011    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca" pod "4bfeda2b-c240-4259-8ac0-0cffa34992de" (UID: "4bfeda2b-c240-4259-8ac0-0cffa34992de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:19.555209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.555195    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfeda2b-c240-4259-8ac0-0cffa34992de-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.556928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.556910    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr" pod "4bfeda2b-c240-4259-8ac0-0cffa34992de" (UID: "4bfeda2b-c240-4259-8ac0-0cffa34992de"). InnerVolumeSpecName "kube-api-access-sdkvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:19.657063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.657026    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkvr\" (UniqueName: \"kubernetes.io/projected/4bfeda2b-c240-4259-8ac0-0cffa34992de-kube-api-access-sdkvr\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.879979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.879881    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"]
Oct 31 00:30:19.880900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880878    5078 cpu_manager.go:414] "containerMap: removing container" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880903    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880925    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880932    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880958    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880966    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880982    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.880989    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.881023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881022    5078 cpu_manager.go:414] "containerMap: removing container" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881029    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881224    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881240    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881249    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5665dbb-ca2f-43f3-b7cd-4f1f1dc257da" containerName="istio-proxy"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881267    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881279    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.881349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.881290    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:19.885748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.885724    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:19.902759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.902729    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"]
Oct 31 00:30:19.960312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.960279    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:19.960312 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.960319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:19.960581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:19.960453    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmnf\" (UniqueName: \"kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.061976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.061935    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.061976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.061973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.062253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.062035    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmnf\" (UniqueName: \"kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.062639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.062616    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.064695 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.064675    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.076605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.076582    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmnf\" (UniqueName: \"kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf\") pod \"maas-api-revoke-sub-ead48221-524q5\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.201439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.201394    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" path="/var/lib/kubelet/pods/223c462f-c0fc-41ad-b18d-60b2dcb4f4ca/volumes"
Oct 31 00:30:20.207391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.207363    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:20.217065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.217037    5078 generic.go:356] "Generic (PLEG): container finished" podID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerID="1c4d88dd9472d4bffd5a6d1bb9205111ae45b82cce4d0c6b102b24826f17d5e2" exitCode=60
Oct 31 00:30:20.217183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.217106    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerDied","Data":"1c4d88dd9472d4bffd5a6d1bb9205111ae45b82cce4d0c6b102b24826f17d5e2"}
Oct 31 00:30:20.217183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.217172    5078 scope.go:122] "RemoveContainer" containerID="e550f8687656f203bb97e23b9d87109774cc6ebe7f8c3714ee119f0fc4f74da4"
Oct 31 00:30:20.217633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.217613    5078 scope.go:122] "RemoveContainer" containerID="1c4d88dd9472d4bffd5a6d1bb9205111ae45b82cce4d0c6b102b24826f17d5e2"
Oct 31 00:30:20.218034 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:20.217985    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc_odh-ai-gateway-infra(12cc1b94-5558-4fc7-8c9d-7838af1454d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7"
Oct 31 00:30:20.222767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.222746    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"
Oct 31 00:30:20.222859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.222769    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj" event={"ID":"4bfeda2b-c240-4259-8ac0-0cffa34992de","Type":"ContainerDied","Data":"1c4904030346b9e3afef96f19a34f850b238b38bf47c3c40412677bccbeac58c"}
Oct 31 00:30:20.258390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.258357    5078 scope.go:122] "RemoveContainer" containerID="4f4bff6f7873fe137e291901e9446c342ba2b43837d6b09dfc4b4a7793fbed1a"
Oct 31 00:30:20.305634 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.305601    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"]
Oct 31 00:30:20.346510 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.346327    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"]
Oct 31 00:30:20.357599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.357570    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7c5da3-29890110-p2lkj"]
Oct 31 00:30:20.532887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.532857    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:20.535135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.535110    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:20.538427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.538404    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"]
Oct 31 00:30:20.538602 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:20.538575    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c732715_1515_4487_8c97_fe8f7566e72f.slice/crio-0439536b1c58db6f2924cfa6f66d07bdd2e28651e043e2a9380e4f5c38fd7acf WatchSource:0}: Error finding container 0439536b1c58db6f2924cfa6f66d07bdd2e28651e043e2a9380e4f5c38fd7acf: Status 404 returned error can't find the container with id 0439536b1c58db6f2924cfa6f66d07bdd2e28651e043e2a9380e4f5c38fd7acf
Oct 31 00:30:20.598516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.597189    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf\" (UniqueName: \"kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf\") pod \"f4ad2eba-5caa-4231-87d5-50431b292d29\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") "
Oct 31 00:30:20.598516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.597987    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca\") pod \"f4ad2eba-5caa-4231-87d5-50431b292d29\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") "
Oct 31 00:30:20.598516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.598439    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca" pod "bc181ef9-ba0b-41fc-9933-4e113f546ce8" (UID: "bc181ef9-ba0b-41fc-9933-4e113f546ce8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:20.598827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.598539    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca\") pod \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") "
Oct 31 00:30:20.598964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.598915    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf\" (UniqueName: \"kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf\") pod \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\" (UID: \"bc181ef9-ba0b-41fc-9933-4e113f546ce8\") "
Oct 31 00:30:20.602796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.599075    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca\") pod \"f4ad2eba-5caa-4231-87d5-50431b292d29\" (UID: \"f4ad2eba-5caa-4231-87d5-50431b292d29\") "
Oct 31 00:30:20.602796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.599922    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc181ef9-ba0b-41fc-9933-4e113f546ce8-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:20.602796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.600373    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca" pod "f4ad2eba-5caa-4231-87d5-50431b292d29" (UID: "f4ad2eba-5caa-4231-87d5-50431b292d29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:20.602796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.601657    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca" pod "f4ad2eba-5caa-4231-87d5-50431b292d29" (UID: "f4ad2eba-5caa-4231-87d5-50431b292d29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:20.603318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.603175    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf" pod "bc181ef9-ba0b-41fc-9933-4e113f546ce8" (UID: "bc181ef9-ba0b-41fc-9933-4e113f546ce8"). InnerVolumeSpecName "kube-api-access-wfcsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:20.606476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.606434    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf" pod "f4ad2eba-5caa-4231-87d5-50431b292d29" (UID: "f4ad2eba-5caa-4231-87d5-50431b292d29"). InnerVolumeSpecName "kube-api-access-vrvvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:20.701138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.701101    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcsf\" (UniqueName: \"kubernetes.io/projected/bc181ef9-ba0b-41fc-9933-4e113f546ce8-kube-api-access-wfcsf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:20.701138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.701137    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:20.701328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.701153    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvvf\" (UniqueName: \"kubernetes.io/projected/f4ad2eba-5caa-4231-87d5-50431b292d29-kube-api-access-vrvvf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:20.701328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.701167    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad2eba-5caa-4231-87d5-50431b292d29-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:20.981629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.981571    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:30:20.981795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:20.981651    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:30:21.233013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.232981    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"
Oct 31 00:30:21.233013 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.233004    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq" event={"ID":"f4ad2eba-5caa-4231-87d5-50431b292d29","Type":"ContainerDied","Data":"1d7852e4b99eb12ec91ac061617dafb45efe3a101166c314850e2fb6a5321855"}
Oct 31 00:30:21.233503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.233052    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d7852e4b99eb12ec91ac061617dafb45efe3a101166c314850e2fb6a5321855"
Oct 31 00:30:21.246276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.246228    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf" event={"ID":"bc181ef9-ba0b-41fc-9933-4e113f546ce8","Type":"ContainerDied","Data":"2d9e9b801a465a5d1b348eb3d8286319df61f9245a8a755d4d2f33f12d780373"}
Oct 31 00:30:21.246516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.246343    5078 scope.go:122] "RemoveContainer" containerID="b087c2d8d321a2e2952003da621cccd2e09d5b82d1b2f589d6a67a6eba3c5fdb"
Oct 31 00:30:21.246717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.246677    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"
Oct 31 00:30:21.251909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.251879    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5" event={"ID":"8c732715-1515-4487-8c97-fe8f7566e72f","Type":"ContainerStarted","Data":"bcd65622019f4ba5a2faffee85aa803f33adc8101476bcdbaa9a335a147846c8"}
Oct 31 00:30:21.252007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.251965    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5" event={"ID":"8c732715-1515-4487-8c97-fe8f7566e72f","Type":"ContainerStarted","Data":"0439536b1c58db6f2924cfa6f66d07bdd2e28651e043e2a9380e4f5c38fd7acf"}
Oct 31 00:30:21.308657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.308626    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"]
Oct 31 00:30:21.312190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.312163    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-945abd-29890110-2z4pf"]
Oct 31 00:30:21.342029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.341969    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5" podStartSLOduration=2.341949221 podStartE2EDuration="2.341949221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:21.341191456 +0000 UTC m=+2189.287562177" watchObservedRunningTime="2026-10-31 00:30:21.341949221 +0000 UTC m=+2189.288319941"
Oct 31 00:30:21.469480 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.469428    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:21.626828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.626792    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca\") pod \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") "
Oct 31 00:30:21.627027 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.626961    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h\" (UniqueName: \"kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h\") pod \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\" (UID: \"12cc1b94-5558-4fc7-8c9d-7838af1454d7\") "
Oct 31 00:30:21.627296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.627257    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca" pod "12cc1b94-5558-4fc7-8c9d-7838af1454d7" (UID: "12cc1b94-5558-4fc7-8c9d-7838af1454d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:21.627537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.627514    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc1b94-5558-4fc7-8c9d-7838af1454d7-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:21.629376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.629338    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h" pod "12cc1b94-5558-4fc7-8c9d-7838af1454d7" (UID: "12cc1b94-5558-4fc7-8c9d-7838af1454d7"). InnerVolumeSpecName "kube-api-access-n9d2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:21.728576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:21.728545    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9d2h\" (UniqueName: \"kubernetes.io/projected/12cc1b94-5558-4fc7-8c9d-7838af1454d7-kube-api-access-n9d2h\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.116540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.116506    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:30:22.213501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.213418    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" path="/var/lib/kubelet/pods/4bfeda2b-c240-4259-8ac0-0cffa34992de/volumes"
Oct 31 00:30:22.214142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.214122    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" path="/var/lib/kubelet/pods/bc181ef9-ba0b-41fc-9933-4e113f546ce8/volumes"
Oct 31 00:30:22.251981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.251956    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.251997    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252059    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252098    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252118    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252276    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252322    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252365    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:22.252820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252451    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252525    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt\") pod \"603f2b77-a5c8-403a-97e9-c143099f8fb3\" (UID: \"603f2b77-a5c8-403a-97e9-c143099f8fb3\") "
Oct 31 00:30:22.252820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252539    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:22.252820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.252661    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:22.253101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.253080    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.253281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.253261    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.253281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.253284    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f2b77-a5c8-403a-97e9-c143099f8fb3-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.253281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.253114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:22.253621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.253232    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:22.255056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.255031    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:22.255260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.255243    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "kube-api-access-2k5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:22.255350 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.255329    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:30:22.255482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.255447    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy" pod "603f2b77-a5c8-403a-97e9-c143099f8fb3" (UID: "603f2b77-a5c8-403a-97e9-c143099f8fb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:22.263137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.263114    5078 generic.go:356] "Generic (PLEG): container finished" podID="8c732715-1515-4487-8c97-fe8f7566e72f" containerID="bcd65622019f4ba5a2faffee85aa803f33adc8101476bcdbaa9a335a147846c8" exitCode=0
Oct 31 00:30:22.263238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.263192    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5" event={"ID":"8c732715-1515-4487-8c97-fe8f7566e72f","Type":"ContainerDied","Data":"bcd65622019f4ba5a2faffee85aa803f33adc8101476bcdbaa9a335a147846c8"}
Oct 31 00:30:22.265157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.265133    5078 generic.go:356] "Generic (PLEG): container finished" podID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerID="a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7" exitCode=0
Oct 31 00:30:22.265245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.265179    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"
Oct 31 00:30:22.265245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.265209    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" event={"ID":"603f2b77-a5c8-403a-97e9-c143099f8fb3","Type":"ContainerDied","Data":"a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7"}
Oct 31 00:30:22.265366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.265261    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z" event={"ID":"603f2b77-a5c8-403a-97e9-c143099f8fb3","Type":"ContainerDied","Data":"9adadc9592a1dc471acc238a2920c582522150a68c3ec583f59674bc4e6bc4c1"}
Oct 31 00:30:22.265366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.265293    5078 scope.go:122] "RemoveContainer" containerID="a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7"
Oct 31 00:30:22.269800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.267017    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc" event={"ID":"12cc1b94-5558-4fc7-8c9d-7838af1454d7","Type":"ContainerDied","Data":"789c2df012fabd05a3af4b1a2ee853d33c00d4e88f283c52457b912407dba2c7"}
Oct 31 00:30:22.269800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.267029    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"
Oct 31 00:30:22.289726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.289706    5078 scope.go:122] "RemoveContainer" containerID="444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"
Oct 31 00:30:22.301110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.301089    5078 scope.go:122] "RemoveContainer" containerID="a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7"
Oct 31 00:30:22.301348 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:22.301328    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7\": container with ID starting with a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7 not found: ID does not exist" containerID="a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7"
Oct 31 00:30:22.301406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.301358    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7"} err="failed to get container status \"a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7\": rpc error: code = NotFound desc = could not find container \"a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7\": container with ID starting with a24cf16a68edc38877ded3803930be98fd235f3ffbeae8f2a1701f3c1cda38b7 not found: ID does not exist"
Oct 31 00:30:22.301406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.301378    5078 scope.go:122] "RemoveContainer" containerID="444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"
Oct 31 00:30:22.301705 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:22.301685    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5\": container with ID starting with 444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5 not found: ID does not exist" containerID="444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"
Oct 31 00:30:22.301763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.301714    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5"} err="failed to get container status \"444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5\": rpc error: code = NotFound desc = could not find container \"444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5\": container with ID starting with 444dbab095ef693cf7adfe424403ba9e585d029f14b37cb27341071edce955d5 not found: ID does not exist"
Oct 31 00:30:22.301763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.301728    5078 scope.go:122] "RemoveContainer" containerID="1c4d88dd9472d4bffd5a6d1bb9205111ae45b82cce4d0c6b102b24826f17d5e2"
Oct 31 00:30:22.314585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.314564    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"]
Oct 31 00:30:22.319753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.319732    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890110-qxqqc"]
Oct 31 00:30:22.341317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.341295    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"]
Oct 31 00:30:22.355094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.355073    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.355094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.355094    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.355289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.355105    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.355289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.355113    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.355289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.355127    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5wt\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-kube-api-access-2k5wt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.355289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.355139    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f2b77-a5c8-403a-97e9-c143099f8fb3-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:22.358145 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.358109    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2d9875-openshift-default-599c97f5cf-xwc5z"]
Oct 31 00:30:22.661378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.661340    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" containerID="cri-o://3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d" gracePeriod=25
Oct 31 00:30:22.702706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.702668    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" containerID="cri-o://470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804" gracePeriod=25
Oct 31 00:30:22.825585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:22.825549    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:23.285320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.285287    5078 generic.go:356] "Generic (PLEG): container finished" podID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerID="faba5592806f102eccc49d8f9f1b7675d239e628fa99db84b139e966000b332b" exitCode=0
Oct 31 00:30:23.285821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.285364    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" event={"ID":"cf087db0-eaae-4c32-8e65-4a6238b2e282","Type":"ContainerDied","Data":"faba5592806f102eccc49d8f9f1b7675d239e628fa99db84b139e966000b332b"}
Oct 31 00:30:23.432511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.427810    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:30:23.584058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.583955    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.584266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.584060    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.584266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.584122    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.584266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.584221    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.584266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.584262    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.585876    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.585933    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.585992    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.586050    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert\") pod \"cf087db0-eaae-4c32-8e65-4a6238b2e282\" (UID: \"cf087db0-eaae-4c32-8e65-4a6238b2e282\") "
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.586890    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.586987    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.587031    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.587239    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:23.587763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.587604    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:23.588593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.588515    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.588593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.588541    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.588593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.588557    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.588593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.588571    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.596127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.596021    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:23.603427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.603389    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:23.603743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.603705    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:30:23.611072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.611030    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl" pod "cf087db0-eaae-4c32-8e65-4a6238b2e282" (UID: "cf087db0-eaae-4c32-8e65-4a6238b2e282"). InnerVolumeSpecName "kube-api-access-fpdcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:23.639901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.639752    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:23.690740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.690669    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf087db0-eaae-4c32-8e65-4a6238b2e282-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.690740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.690721    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.690740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.690740    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.691100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.690755    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpdcl\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-kube-api-access-fpdcl\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.691100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.690772    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf087db0-eaae-4c32-8e65-4a6238b2e282-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.792270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.792241    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf\" (UniqueName: \"kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf\") pod \"8c732715-1515-4487-8c97-fe8f7566e72f\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") "
Oct 31 00:30:23.792437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.792395    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca\") pod \"8c732715-1515-4487-8c97-fe8f7566e72f\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") "
Oct 31 00:30:23.792437 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.792423    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca\") pod \"8c732715-1515-4487-8c97-fe8f7566e72f\" (UID: \"8c732715-1515-4487-8c97-fe8f7566e72f\") "
Oct 31 00:30:23.792897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.792864    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca" pod "8c732715-1515-4487-8c97-fe8f7566e72f" (UID: "8c732715-1515-4487-8c97-fe8f7566e72f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:23.794678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.794651    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf" pod "8c732715-1515-4487-8c97-fe8f7566e72f" (UID: "8c732715-1515-4487-8c97-fe8f7566e72f"). InnerVolumeSpecName "kube-api-access-tcmnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:23.794813 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.794754    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca" pod "8c732715-1515-4487-8c97-fe8f7566e72f" (UID: "8c732715-1515-4487-8c97-fe8f7566e72f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:23.894289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.894254    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmnf\" (UniqueName: \"kubernetes.io/projected/8c732715-1515-4487-8c97-fe8f7566e72f-kube-api-access-tcmnf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.894507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.894293    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:23.894507 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:23.894310    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c732715-1515-4487-8c97-fe8f7566e72f-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:24.200798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.200680    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" path="/var/lib/kubelet/pods/12cc1b94-5558-4fc7-8c9d-7838af1454d7/volumes"
Oct 31 00:30:24.201290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.201266    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" path="/var/lib/kubelet/pods/603f2b77-a5c8-403a-97e9-c143099f8fb3/volumes"
Oct 31 00:30:24.296818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.296788    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"
Oct 31 00:30:24.297268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.296789    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5" event={"ID":"8c732715-1515-4487-8c97-fe8f7566e72f","Type":"ContainerDied","Data":"0439536b1c58db6f2924cfa6f66d07bdd2e28651e043e2a9380e4f5c38fd7acf"}
Oct 31 00:30:24.297268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.296935    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0439536b1c58db6f2924cfa6f66d07bdd2e28651e043e2a9380e4f5c38fd7acf"
Oct 31 00:30:24.298940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.298903    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5" event={"ID":"cf087db0-eaae-4c32-8e65-4a6238b2e282","Type":"ContainerDied","Data":"fff985c9dadcc3cf20e28778344b3722fa112b688ce9b492f87878df1202a584"}
Oct 31 00:30:24.299087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.298962    5078 scope.go:122] "RemoveContainer" containerID="faba5592806f102eccc49d8f9f1b7675d239e628fa99db84b139e966000b332b"
Oct 31 00:30:24.299087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.299068    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"
Oct 31 00:30:24.336076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.336042    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"]
Oct 31 00:30:24.342319 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:24.342295    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nbhq5"]
Oct 31 00:30:26.197851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:26.197805    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" path="/var/lib/kubelet/pods/cf087db0-eaae-4c32-8e65-4a6238b2e282/volumes"
Oct 31 00:30:27.028368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.028332    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"]
Oct 31 00:30:27.029287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029259    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:27.029287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029287    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfeda2b-c240-4259-8ac0-0cffa34992de" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029311    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8c732715-1515-4487-8c97-fe8f7566e72f" containerName="revoke-keys"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029318    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c732715-1515-4487-8c97-fe8f7566e72f" containerName="revoke-keys"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029329    5078 cpu_manager.go:414] "containerMap: removing container" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029336    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029361    5078 cpu_manager.go:414] "containerMap: removing container" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029369    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029384    5078 cpu_manager.go:414] "containerMap: removing container" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029391    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029405    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f4ad2eba-5caa-4231-87d5-50431b292d29" containerName="revoke-keys"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029411    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ad2eba-5caa-4231-87d5-50431b292d29" containerName="revoke-keys"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029435    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029443    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029475    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029483    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029505    5078 cpu_manager.go:414] "containerMap: removing container" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029512    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029523    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029530    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.029551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029562    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029569    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029589    5078 cpu_manager.go:414] "containerMap: removing container" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029596    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029750    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029763    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029777    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c732715-1515-4487-8c97-fe8f7566e72f" containerName="revoke-keys"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029788    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029799    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029807    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="223c462f-c0fc-41ad-b18d-60b2dcb4f4ca" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029818    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ad2eba-5caa-4231-87d5-50431b292d29" containerName="revoke-keys"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029831    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf087db0-eaae-4c32-8e65-4a6238b2e282" containerName="istio-proxy"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029842    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.030166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.029851    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.035526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.035500    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"]
Oct 31 00:30:27.035720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.035595    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.036513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.036304    5078 cpu_manager.go:414] "containerMap: removing container" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.036513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.036323    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cc1b94-5558-4fc7-8c9d-7838af1454d7" containerName="cleanup"
Oct 31 00:30:27.036713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.036592    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc181ef9-ba0b-41fc-9933-4e113f546ce8" containerName="cleanup"
Oct 31 00:30:27.036713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.036611    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="603f2b77-a5c8-403a-97e9-c143099f8fb3" containerName="istio-proxy"
Oct 31 00:30:27.041098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.041073    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.043619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.043595    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:30:27.043727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.043636    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:27.043727 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.043664    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"]
Oct 31 00:30:27.049797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.049773    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"]
Oct 31 00:30:27.138555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.138525    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.138703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.138582    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9tq\" (UniqueName: \"kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.138763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.138719    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.239949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.239915    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.240349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.239963    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdjt\" (UniqueName: \"kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.240349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.240001    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.240349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.240249    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.240498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.240379    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.240498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.240485    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9tq\" (UniqueName: \"kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.241144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.241117    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.242554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.242530    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.273827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.273802    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9tq\" (UniqueName: \"kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq\") pod \"maas-api-revoke-sub-e853c3af-xccvc\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.342666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.342569    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdjt\" (UniqueName: \"kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.342666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.342617    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.342904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.342727    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.343760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.343728    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.345515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.345456    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.362194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.362169    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:27.386966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.386938    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdjt\" (UniqueName: \"kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt\") pod \"maas-api-revoke-sub-eedb204d-84xql\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.664608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.664577    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"]
Oct 31 00:30:27.669417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.668359    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:27.669417 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:27.668744    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdf4c03_7887_4170_bf7d_394732ef30dd.slice/crio-c3141f429b97857075c08fd68cf83c70494f0a6046216f9266e227f14245a9a6 WatchSource:0}: Error finding container c3141f429b97857075c08fd68cf83c70494f0a6046216f9266e227f14245a9a6: Status 404 returned error can't find the container with id c3141f429b97857075c08fd68cf83c70494f0a6046216f9266e227f14245a9a6
Oct 31 00:30:27.990280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:27.988349    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"]
Oct 31 00:30:27.990280 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:27.990117    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea3fe1_8dc8_4c63_baf5_5891e1a0268e.slice/crio-dc7cd393a4cc398a2968f92fb79c27949fbb270c565bb97f193cb2b08a8b1039 WatchSource:0}: Error finding container dc7cd393a4cc398a2968f92fb79c27949fbb270c565bb97f193cb2b08a8b1039: Status 404 returned error can't find the container with id dc7cd393a4cc398a2968f92fb79c27949fbb270c565bb97f193cb2b08a8b1039
Oct 31 00:30:28.233656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.233624    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"]
Oct 31 00:30:28.233876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.233854    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" podUID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" containerName="maas-api" containerID="cri-o://34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25" gracePeriod=30
Oct 31 00:30:28.343447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.343394    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql" event={"ID":"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e","Type":"ContainerStarted","Data":"dc7cd393a4cc398a2968f92fb79c27949fbb270c565bb97f193cb2b08a8b1039"}
Oct 31 00:30:28.347262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.347213    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc" event={"ID":"dcdf4c03-7887-4170-bf7d-394732ef30dd","Type":"ContainerStarted","Data":"4a5bd963d2e81e0f50f126a5ccab0db1a255da61e03dcc807371114c974b1175"}
Oct 31 00:30:28.347419 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.347301    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc" event={"ID":"dcdf4c03-7887-4170-bf7d-394732ef30dd","Type":"ContainerStarted","Data":"c3141f429b97857075c08fd68cf83c70494f0a6046216f9266e227f14245a9a6"}
Oct 31 00:30:28.375023 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.374965    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc" podStartSLOduration=1.3749503330000001 podStartE2EDuration="1.374950333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:28.371605983 +0000 UTC m=+2196.317976703" watchObservedRunningTime="2026-10-31 00:30:28.374950333 +0000 UTC m=+2196.321321057"
Oct 31 00:30:28.387503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.387187    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" podUID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.214:8443/health\": dial tcp 10.132.0.214:8443: connect: connection refused"
Oct 31 00:30:28.653687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.653660    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:30:28.719354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.719247    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca\") pod \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") "
Oct 31 00:30:28.719575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.719402    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb\" (UniqueName: \"kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb\") pod \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") "
Oct 31 00:30:28.719649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.719597    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\") pod \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") "
Oct 31 00:30:28.719649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.719632    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs\") pod \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\" (UID: \"8c7f7144-264c-4b10-ad57-abbe1bba8fef\") "
Oct 31 00:30:28.719754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.719654    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca" pod "8c7f7144-264c-4b10-ad57-abbe1bba8fef" (UID: "8c7f7144-264c-4b10-ad57-abbe1bba8fef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:28.720194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.720038    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7144-264c-4b10-ad57-abbe1bba8fef-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:28.722338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.722312    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs" pod "8c7f7144-264c-4b10-ad57-abbe1bba8fef" (UID: "8c7f7144-264c-4b10-ad57-abbe1bba8fef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:28.722977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.722955    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls" pod "8c7f7144-264c-4b10-ad57-abbe1bba8fef" (UID: "8c7f7144-264c-4b10-ad57-abbe1bba8fef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:28.723119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.723103    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb" pod "8c7f7144-264c-4b10-ad57-abbe1bba8fef" (UID: "8c7f7144-264c-4b10-ad57-abbe1bba8fef"). InnerVolumeSpecName "kube-api-access-kbwtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:28.745672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.745396    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:28.775823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.775795    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:28.821979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.821937    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbwtb\" (UniqueName: \"kubernetes.io/projected/8c7f7144-264c-4b10-ad57-abbe1bba8fef-kube-api-access-kbwtb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:28.822287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.822236    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:28.822287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:28.822264    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7144-264c-4b10-ad57-abbe1bba8fef-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:29.360009 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.359973    5078 generic.go:356] "Generic (PLEG): container finished" podID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" containerID="34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25" exitCode=0
Oct 31 00:30:29.360488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.360020    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"
Oct 31 00:30:29.360488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.360049    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" event={"ID":"8c7f7144-264c-4b10-ad57-abbe1bba8fef","Type":"ContainerDied","Data":"34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25"}
Oct 31 00:30:29.360488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.360114    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr" event={"ID":"8c7f7144-264c-4b10-ad57-abbe1bba8fef","Type":"ContainerDied","Data":"52d3c6862fbd7242d81ad9a53634c13ef81d248c98a9e7ca31ad3d74fd233e92"}
Oct 31 00:30:29.360488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.360143    5078 scope.go:122] "RemoveContainer" containerID="34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25"
Oct 31 00:30:29.362578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.362551    5078 generic.go:356] "Generic (PLEG): container finished" podID="7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" containerID="9d3d193131631eb40006edcd3f116abfcaaf9ab6d9825cece92886b82d6faaba" exitCode=0
Oct 31 00:30:29.362959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.362637    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql" event={"ID":"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e","Type":"ContainerDied","Data":"9d3d193131631eb40006edcd3f116abfcaaf9ab6d9825cece92886b82d6faaba"}
Oct 31 00:30:29.365891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.365862    5078 generic.go:356] "Generic (PLEG): container finished" podID="dcdf4c03-7887-4170-bf7d-394732ef30dd" containerID="4a5bd963d2e81e0f50f126a5ccab0db1a255da61e03dcc807371114c974b1175" exitCode=0
Oct 31 00:30:29.366019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.366000    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc" event={"ID":"dcdf4c03-7887-4170-bf7d-394732ef30dd","Type":"ContainerDied","Data":"4a5bd963d2e81e0f50f126a5ccab0db1a255da61e03dcc807371114c974b1175"}
Oct 31 00:30:29.374915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.374895    5078 scope.go:122] "RemoveContainer" containerID="34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25"
Oct 31 00:30:29.375241 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:29.375216    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25\": container with ID starting with 34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25 not found: ID does not exist" containerID="34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25"
Oct 31 00:30:29.375325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.375252    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25"} err="failed to get container status \"34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25\": rpc error: code = NotFound desc = could not find container \"34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25\": container with ID starting with 34824c9f8df54648ead45994b01ee233a81c76f6d5172b3bf4d4fe9f5f5dde25 not found: ID does not exist"
Oct 31 00:30:29.432175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.432089    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"]
Oct 31 00:30:29.435703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.435677    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d4ad9-65fcd8dffd-fq9jr"]
Oct 31 00:30:29.857219 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:29.855078    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main" containerID="cri-o://ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b" gracePeriod=45
Oct 31 00:30:30.037834 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:30.037679    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b68920b_68ef_4980_a743_f46eddf82cf5.slice/crio-ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b68920b_68ef_4980_a743_f46eddf82cf5.slice/crio-conmon-ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:30:30.231388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.226770    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:30:30.231388 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.228248    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" path="/var/lib/kubelet/pods/8c7f7144-264c-4b10-ad57-abbe1bba8fef/volumes"
Oct 31 00:30:30.234233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.234205    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"]
Oct 31 00:30:30.235095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.235068    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing"
Oct 31 00:30:30.235212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.235096    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing"
Oct 31 00:30:30.235212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.235188    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" containerName="maas-api"
Oct 31 00:30:30.235212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.235197    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" containerName="maas-api"
Oct 31 00:30:30.235394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.235366    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7f7144-264c-4b10-ad57-abbe1bba8fef" containerName="maas-api"
Oct 31 00:30:30.235394 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.235388    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerName="payload-processing"
Oct 31 00:30:30.241777 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.241517    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.261972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.261949    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"]
Oct 31 00:30:30.280990 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.280965    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:30:30.363358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.363321    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home\") pod \"9b68920b-68ef-4980-a743-f46eddf82cf5\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") "
Oct 31 00:30:30.364066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364039    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume\") pod \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") "
Oct 31 00:30:30.364269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364254    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache\") pod \"9b68920b-68ef-4980-a743-f46eddf82cf5\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") "
Oct 31 00:30:30.364409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364394    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p\" (UniqueName: \"kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p\") pod \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\" (UID: \"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4\") "
Oct 31 00:30:30.364568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364553    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd\" (UniqueName: \"kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd\") pod \"9b68920b-68ef-4980-a743-f46eddf82cf5\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") "
Oct 31 00:30:30.364698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364685    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir\") pod \"9b68920b-68ef-4980-a743-f46eddf82cf5\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") "
Oct 31 00:30:30.364844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364829    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs\") pod \"9b68920b-68ef-4980-a743-f46eddf82cf5\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") "
Oct 31 00:30:30.364971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.364956    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm\") pod \"9b68920b-68ef-4980-a743-f46eddf82cf5\" (UID: \"9b68920b-68ef-4980-a743-f46eddf82cf5\") "
Oct 31 00:30:30.365609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.365583    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.365838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.365819    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.365957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.365943    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7r6\" (UniqueName: \"kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.366476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.366436    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home" pod "9b68920b-68ef-4980-a743-f46eddf82cf5" (UID: "9b68920b-68ef-4980-a743-f46eddf82cf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:30.368436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.368411    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume" pod "6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" (UID: "6b84a98f-261e-4eb0-bdc3-ab5c60b200c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.369203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.369182    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache" pod "9b68920b-68ef-4980-a743-f46eddf82cf5" (UID: "9b68920b-68ef-4980-a743-f46eddf82cf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:30.376515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.376479    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir" pod "9b68920b-68ef-4980-a743-f46eddf82cf5" (UID: "9b68920b-68ef-4980-a743-f46eddf82cf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:30.393557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.393356    5078 generic.go:356] "Generic (PLEG): container finished" podID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerID="ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b" exitCode=0
Oct 31 00:30:30.393557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.393406    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"
Oct 31 00:30:30.393934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.393755    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" event={"ID":"9b68920b-68ef-4980-a743-f46eddf82cf5","Type":"ContainerDied","Data":"ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b"}
Oct 31 00:30:30.393934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.393801    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw" event={"ID":"9b68920b-68ef-4980-a743-f46eddf82cf5","Type":"ContainerDied","Data":"aed8aa631bce388726da4f6fdbd5c770a8775b6778bfce806f7cf35931e36782"}
Oct 31 00:30:30.393934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.393825    5078 scope.go:122] "RemoveContainer" containerID="ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b"
Oct 31 00:30:30.396773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.396300    5078 generic.go:356] "Generic (PLEG): container finished" podID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" containerID="42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231" exitCode=0
Oct 31 00:30:30.396773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.396366    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"
Oct 31 00:30:30.396773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.396429    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" event={"ID":"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4","Type":"ContainerDied","Data":"42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231"}
Oct 31 00:30:30.397382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.397337    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p" pod "6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" (UID: "6b84a98f-261e-4eb0-bdc3-ab5c60b200c4"). InnerVolumeSpecName "kube-api-access-2vc7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:30.397572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.397537    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z" event={"ID":"6b84a98f-261e-4eb0-bdc3-ab5c60b200c4","Type":"ContainerDied","Data":"857542eb30ec342a3234d379a0df1dd06e92e5b9e9fe2670d3d6e204616bb720"}
Oct 31 00:30:30.398587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.398341    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm" pod "9b68920b-68ef-4980-a743-f46eddf82cf5" (UID: "9b68920b-68ef-4980-a743-f46eddf82cf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:30.398854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.398823    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs" pod "9b68920b-68ef-4980-a743-f46eddf82cf5" (UID: "9b68920b-68ef-4980-a743-f46eddf82cf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:30.399046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.399027    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd" pod "9b68920b-68ef-4980-a743-f46eddf82cf5" (UID: "9b68920b-68ef-4980-a743-f46eddf82cf5"). InnerVolumeSpecName "kube-api-access-7hhjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:30.436671 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.436070    5078 scope.go:122] "RemoveContainer" containerID="ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b"
Oct 31 00:30:30.436671 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:30.436584    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b\": container with ID starting with ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b not found: ID does not exist" containerID="ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b"
Oct 31 00:30:30.436671 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.436616    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b"} err="failed to get container status \"ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b\": rpc error: code = NotFound desc = could not find container \"ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b\": container with ID starting with ab699e6eb7de49d39fdd47f5b0dd851c56ea149b8df2bfb7e695e4ca2991293b not found: ID does not exist"
Oct 31 00:30:30.436671 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.436637    5078 scope.go:122] "RemoveContainer" containerID="42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231"
Oct 31 00:30:30.451622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.451596    5078 scope.go:122] "RemoveContainer" containerID="42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231"
Oct 31 00:30:30.451931 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:30.451908    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231\": container with ID starting with 42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231 not found: ID does not exist" containerID="42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231"
Oct 31 00:30:30.452209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.451940    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231"} err="failed to get container status \"42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231\": rpc error: code = NotFound desc = could not find container \"42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231\": container with ID starting with 42f148ff7dbf04edebe9f517557dc67f3d8742bacc3d427656dd6cf7acf19231 not found: ID does not exist"
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.467900    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468037    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468072    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7r6\" (UniqueName: \"kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468365    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hhjd\" (UniqueName: \"kubernetes.io/projected/9b68920b-68ef-4980-a743-f46eddf82cf5-kube-api-access-7hhjd\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468392    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468409    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b68920b-68ef-4980-a743-f46eddf82cf5-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468422    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468501    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468516    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468531    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b68920b-68ef-4980-a743-f46eddf82cf5-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.468599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.468547    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vc7p\" (UniqueName: \"kubernetes.io/projected/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4-kube-api-access-2vc7p\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.469904 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.469876    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.473771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.473720    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.495707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.495647    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7r6\" (UniqueName: \"kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.604599 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.603439    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:30.607060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.606903    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:30.674492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.669801    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:30.683362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.681026    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt\" (UniqueName: \"kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt\") pod \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") "
Oct 31 00:30:30.683362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.681443    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca\") pod \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") "
Oct 31 00:30:30.683362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.681642    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca\") pod \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\" (UID: \"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e\") "
Oct 31 00:30:30.683362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.681977    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca" pod "7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" (UID: "7eea3fe1-8dc8-4c63-baf5-5891e1a0268e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.683362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.682505    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.685328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.685293    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca" pod "7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" (UID: "7eea3fe1-8dc8-4c63-baf5-5891e1a0268e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:30.703196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.702663    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt" pod "7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" (UID: "7eea3fe1-8dc8-4c63-baf5-5891e1a0268e"). InnerVolumeSpecName "kube-api-access-5jdjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:30.762740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.762711    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"]
Oct 31 00:30:30.769940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.769914    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-w0-c60826-kserve-8497468c4d-s54tw"]
Oct 31 00:30:30.783014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.782985    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq\" (UniqueName: \"kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq\") pod \"dcdf4c03-7887-4170-bf7d-394732ef30dd\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") "
Oct 31 00:30:30.783261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.783235    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca\") pod \"dcdf4c03-7887-4170-bf7d-394732ef30dd\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") "
Oct 31 00:30:30.783473 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.783441    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca\") pod \"dcdf4c03-7887-4170-bf7d-394732ef30dd\" (UID: \"dcdf4c03-7887-4170-bf7d-394732ef30dd\") "
Oct 31 00:30:30.784059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.784042    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.784248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.784064    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdjt\" (UniqueName: \"kubernetes.io/projected/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e-kube-api-access-5jdjt\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.784992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.784953    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca" pod "dcdf4c03-7887-4170-bf7d-394732ef30dd" (UID: "dcdf4c03-7887-4170-bf7d-394732ef30dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.789665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.789627    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq" pod "dcdf4c03-7887-4170-bf7d-394732ef30dd" (UID: "dcdf4c03-7887-4170-bf7d-394732ef30dd"). InnerVolumeSpecName "kube-api-access-pj9tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:30.794861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.794823    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca" pod "dcdf4c03-7887-4170-bf7d-394732ef30dd" (UID: "dcdf4c03-7887-4170-bf7d-394732ef30dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:30.802644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.802605    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"]
Oct 31 00:30:30.806294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.806249    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d91de6-5cd5f9574f-jnc5z"]
Oct 31 00:30:30.920532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.920355    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9tq\" (UniqueName: \"kubernetes.io/projected/dcdf4c03-7887-4170-bf7d-394732ef30dd-kube-api-access-pj9tq\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.920532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.920391    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.920532 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.920408    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdf4c03-7887-4170-bf7d-394732ef30dd-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.927886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.927849    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"]
Oct 31 00:30:30.928131 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.928102    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy" containerID="cri-o://fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd" gracePeriod=30
Oct 31 00:30:30.981017 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:30.980988    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a20eb05_3297_44ff_88f4_11a2e88ba19d.slice/crio-228f7017dbf9c65a618510072ac40df2af131db0764564408ebf29b959f8091a WatchSource:0}: Error finding container 228f7017dbf9c65a618510072ac40df2af131db0764564408ebf29b959f8091a: Status 404 returned error can't find the container with id 228f7017dbf9c65a618510072ac40df2af131db0764564408ebf29b959f8091a
Oct 31 00:30:30.981664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:30.981616    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"]
Oct 31 00:30:31.407042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.407005    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql" event={"ID":"7eea3fe1-8dc8-4c63-baf5-5891e1a0268e","Type":"ContainerDied","Data":"dc7cd393a4cc398a2968f92fb79c27949fbb270c565bb97f193cb2b08a8b1039"}
Oct 31 00:30:31.407515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.407067    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7cd393a4cc398a2968f92fb79c27949fbb270c565bb97f193cb2b08a8b1039"
Oct 31 00:30:31.407515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.407012    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"
Oct 31 00:30:31.408720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.408700    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"
Oct 31 00:30:31.408829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.408725    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc" event={"ID":"dcdf4c03-7887-4170-bf7d-394732ef30dd","Type":"ContainerDied","Data":"c3141f429b97857075c08fd68cf83c70494f0a6046216f9266e227f14245a9a6"}
Oct 31 00:30:31.408829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.408772    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3141f429b97857075c08fd68cf83c70494f0a6046216f9266e227f14245a9a6"
Oct 31 00:30:31.410232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:31.410207    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4" event={"ID":"5a20eb05-3297-44ff-88f4-11a2e88ba19d","Type":"ContainerStarted","Data":"228f7017dbf9c65a618510072ac40df2af131db0764564408ebf29b959f8091a"}
Oct 31 00:30:32.200357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:32.200324    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b84a98f-261e-4eb0-bdc3-ab5c60b200c4" path="/var/lib/kubelet/pods/6b84a98f-261e-4eb0-bdc3-ab5c60b200c4/volumes"
Oct 31 00:30:32.200903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:32.200881    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" path="/var/lib/kubelet/pods/9b68920b-68ef-4980-a743-f46eddf82cf5/volumes"
Oct 31 00:30:32.418998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:32.418961    5078 generic.go:356] "Generic (PLEG): container finished" podID="5a20eb05-3297-44ff-88f4-11a2e88ba19d" containerID="084678dc3da9fb328054e4646a29117ac83cafb3812578714d0dd6dce0993d56" exitCode=0
Oct 31 00:30:32.419518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:32.419052    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4" event={"ID":"5a20eb05-3297-44ff-88f4-11a2e88ba19d","Type":"ContainerDied","Data":"084678dc3da9fb328054e4646a29117ac83cafb3812578714d0dd6dce0993d56"}
Oct 31 00:30:33.605601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.605574    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:33.770354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.770271    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6\" (UniqueName: \"kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6\") pod \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") "
Oct 31 00:30:33.770544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.770437    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca\") pod \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") "
Oct 31 00:30:33.770607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.770587    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca\") pod \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\" (UID: \"5a20eb05-3297-44ff-88f4-11a2e88ba19d\") "
Oct 31 00:30:33.770855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.770831    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca" pod "5a20eb05-3297-44ff-88f4-11a2e88ba19d" (UID: "5a20eb05-3297-44ff-88f4-11a2e88ba19d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:33.771074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.771056    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:33.772888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.772852    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6" pod "5a20eb05-3297-44ff-88f4-11a2e88ba19d" (UID: "5a20eb05-3297-44ff-88f4-11a2e88ba19d"). InnerVolumeSpecName "kube-api-access-xw7r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:33.773232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.773201    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca" pod "5a20eb05-3297-44ff-88f4-11a2e88ba19d" (UID: "5a20eb05-3297-44ff-88f4-11a2e88ba19d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:33.872518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.872484    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a20eb05-3297-44ff-88f4-11a2e88ba19d-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:33.872518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:33.872513    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7r6\" (UniqueName: \"kubernetes.io/projected/5a20eb05-3297-44ff-88f4-11a2e88ba19d-kube-api-access-xw7r6\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:34.435915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:34.435886    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"
Oct 31 00:30:34.436111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:34.435917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4" event={"ID":"5a20eb05-3297-44ff-88f4-11a2e88ba19d","Type":"ContainerDied","Data":"228f7017dbf9c65a618510072ac40df2af131db0764564408ebf29b959f8091a"}
Oct 31 00:30:34.436111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:34.435965    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228f7017dbf9c65a618510072ac40df2af131db0764564408ebf29b959f8091a"
Oct 31 00:30:36.235653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.235629    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:30:36.390799 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:36.390762    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:30:36.411412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411381    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411516    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411569    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411624    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411690    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411735    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:36.411816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411754    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411811    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:36.411859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411832    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411879    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.411943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.411928    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo\") pod \"75bc669f-f382-41b1-8a91-b79e0b976575\" (UID: \"75bc669f-f382-41b1-8a91-b79e0b976575\") "
Oct 31 00:30:36.412102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412073    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:36.412440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412412    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:36.412590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412529    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.412590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412551    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.412590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412565    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.412590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412579    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc669f-f382-41b1-8a91-b79e0b976575-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.412920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.412875    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:36.414385 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.414355    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:36.415077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.415042    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "kube-api-access-q6qm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:36.415177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.415147    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:36.415368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.415343    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo" pod "75bc669f-f382-41b1-8a91-b79e0b976575" (UID: "75bc669f-f382-41b1-8a91-b79e0b976575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:30:36.452073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.452049    5078 generic.go:356] "Generic (PLEG): container finished" podID="75bc669f-f382-41b1-8a91-b79e0b976575" containerID="fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd" exitCode=0
Oct 31 00:30:36.452191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.452098    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"
Oct 31 00:30:36.452191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.452135    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" event={"ID":"75bc669f-f382-41b1-8a91-b79e0b976575","Type":"ContainerDied","Data":"fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd"}
Oct 31 00:30:36.452307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.452204    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc" event={"ID":"75bc669f-f382-41b1-8a91-b79e0b976575","Type":"ContainerDied","Data":"117bc27024f20a763bb0bda76b4d3ac812aa9eec01a375ed98c20bae40370f0f"}
Oct 31 00:30:36.452307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.452238    5078 scope.go:122] "RemoveContainer" containerID="fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd"
Oct 31 00:30:36.464817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.464538    5078 scope.go:122] "RemoveContainer" containerID="fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd"
Oct 31 00:30:36.464870 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:36.464832    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd\": container with ID starting with fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd not found: ID does not exist" containerID="fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd"
Oct 31 00:30:36.464870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.464854    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd"} err="failed to get container status \"fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd\": rpc error: code = NotFound desc = could not find container \"fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd\": container with ID starting with fadfbf0c1bad59a2d9c2679b73db52f06e132a2036054d84f8ec490009532ecd not found: ID does not exist"
Oct 31 00:30:36.487943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.487920    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"]
Oct 31 00:30:36.493270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.493247    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-zn2fc"]
Oct 31 00:30:36.513620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.513591    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qm8\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-kube-api-access-q6qm8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.513721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.513621    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.513721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.513637    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc669f-f382-41b1-8a91-b79e0b976575-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.513721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.513650    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc669f-f382-41b1-8a91-b79e0b976575-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:36.513721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:36.513663    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc669f-f382-41b1-8a91-b79e0b976575-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:37.546207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:37.546172    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"]
Oct 31 00:30:37.547077 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:37.547037    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy" containerID="cri-o://c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7" gracePeriod=30
Oct 31 00:30:38.197589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.197555    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" path="/var/lib/kubelet/pods/75bc669f-f382-41b1-8a91-b79e0b976575/volumes"
Oct 31 00:30:38.298678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.298643    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"]
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299832    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299857    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299881    5078 cpu_manager.go:414] "containerMap: removing container" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299890    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299905    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dcdf4c03-7887-4170-bf7d-394732ef30dd" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299913    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdf4c03-7887-4170-bf7d-394732ef30dd" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299946    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299952    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299989    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5a20eb05-3297-44ff-88f4-11a2e88ba19d" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.299995    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a20eb05-3297-44ff-88f4-11a2e88ba19d" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.300160    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b68920b-68ef-4980-a743-f46eddf82cf5" containerName="main"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.300175    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bc669f-f382-41b1-8a91-b79e0b976575" containerName="istio-proxy"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.300185    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdf4c03-7887-4170-bf7d-394732ef30dd" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.300204    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" containerName="revoke-keys"
Oct 31 00:30:38.301000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.300219    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a20eb05-3297-44ff-88f4-11a2e88ba19d" containerName="revoke-keys"
Oct 31 00:30:38.321264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.321207    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.322628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.322039    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"]
Oct 31 00:30:38.338749 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.338723    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.338864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.338810    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cg75\" (UniqueName: \"kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.338864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.338862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.362242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.362214    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:30:38.362354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.362289    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:38.440143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.440105    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.440342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.440271    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.440423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.440365    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cg75\" (UniqueName: \"kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.440870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.440844    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.442911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.442887    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.455344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.455293    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cg75\" (UniqueName: \"kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75\") pod \"maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.643908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.643874    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:38.747335 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.746996    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:38.775983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.775944    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:38.948050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:38.948019    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"]
Oct 31 00:30:39.492479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:39.492189    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c" event={"ID":"9d904c7e-e1c8-41fa-9397-3691813d172e","Type":"ContainerStarted","Data":"15af02a777eee73c3f7a0c740b3be12358ba6212e2f9a4b5f406555ddfaf1676"}
Oct 31 00:30:39.492479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:39.492319    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c" event={"ID":"9d904c7e-e1c8-41fa-9397-3691813d172e","Type":"ContainerStarted","Data":"3939b77e2498485123ae1766b9e6241e595c62ca0701b321c2824b8e5aa377eb"}
Oct 31 00:30:39.528838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:39.528775    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c" podStartSLOduration=1.528755196 podStartE2EDuration="1.528755196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:39.527772378 +0000 UTC m=+2207.474143095" watchObservedRunningTime="2026-10-31 00:30:39.528755196 +0000 UTC m=+2207.475125914"
Oct 31 00:30:40.569656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:40.569553    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c" event={"ID":"9d904c7e-e1c8-41fa-9397-3691813d172e","Type":"ContainerDied","Data":"15af02a777eee73c3f7a0c740b3be12358ba6212e2f9a4b5f406555ddfaf1676"}
Oct 31 00:30:40.573535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:40.573501    5078 generic.go:356] "Generic (PLEG): container finished" podID="9d904c7e-e1c8-41fa-9397-3691813d172e" containerID="15af02a777eee73c3f7a0c740b3be12358ba6212e2f9a4b5f406555ddfaf1676" exitCode=0
Oct 31 00:30:41.370684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.370629    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:30:41.371034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.370981    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:30:41.387568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.387538    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:30:41.387752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.387730    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:30:41.502585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.502546    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:30:41.502874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.502841    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:30:41.771550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.771512    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"]
Oct 31 00:30:41.818014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:41.817901    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1c9660-j69n4"]
Oct 31 00:30:42.206224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.200909    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a20eb05-3297-44ff-88f4-11a2e88ba19d" path="/var/lib/kubelet/pods/5a20eb05-3297-44ff-88f4-11a2e88ba19d/volumes"
Oct 31 00:30:42.206224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.203554    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:42.234597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.234566    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75\" (UniqueName: \"kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75\") pod \"9d904c7e-e1c8-41fa-9397-3691813d172e\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") "
Oct 31 00:30:42.234785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.234682    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca\") pod \"9d904c7e-e1c8-41fa-9397-3691813d172e\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") "
Oct 31 00:30:42.234850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.234782    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca\") pod \"9d904c7e-e1c8-41fa-9397-3691813d172e\" (UID: \"9d904c7e-e1c8-41fa-9397-3691813d172e\") "
Oct 31 00:30:42.237549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.237341    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca" pod "9d904c7e-e1c8-41fa-9397-3691813d172e" (UID: "9d904c7e-e1c8-41fa-9397-3691813d172e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:42.239874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.239846    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75" pod "9d904c7e-e1c8-41fa-9397-3691813d172e" (UID: "9d904c7e-e1c8-41fa-9397-3691813d172e"). InnerVolumeSpecName "kube-api-access-7cg75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:42.242880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.242853    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca" pod "9d904c7e-e1c8-41fa-9397-3691813d172e" (UID: "9d904c7e-e1c8-41fa-9397-3691813d172e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:42.337037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.336949    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cg75\" (UniqueName: \"kubernetes.io/projected/9d904c7e-e1c8-41fa-9397-3691813d172e-kube-api-access-7cg75\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:42.337037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.336990    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:42.337037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.337006    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d904c7e-e1c8-41fa-9397-3691813d172e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:42.626763 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.626731    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"
Oct 31 00:30:42.626930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.626764    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c" event={"ID":"9d904c7e-e1c8-41fa-9397-3691813d172e","Type":"ContainerDied","Data":"3939b77e2498485123ae1766b9e6241e595c62ca0701b321c2824b8e5aa377eb"}
Oct 31 00:30:42.626930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.626821    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3939b77e2498485123ae1766b9e6241e595c62ca0701b321c2824b8e5aa377eb"
Oct 31 00:30:42.841752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.841590    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.947931    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.947995    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948059    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948088    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948172    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948282    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948352    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948400    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.952590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.948438    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token\") pod \"9f6de8b3-0e20-48c9-8843-cf574e833566\" (UID: \"9f6de8b3-0e20-48c9-8843-cf574e833566\") "
Oct 31 00:30:42.953908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.953807    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:42.954488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.954229    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:42.954823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.954801    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:42.954957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.954806    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:42.955352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.955315    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:42.956382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.956359    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:42.961515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.961478    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:30:42.961617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.961589    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "kube-api-access-mgl48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:42.963553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:42.963443    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy" pod "9f6de8b3-0e20-48c9-8843-cf574e833566" (UID: "9f6de8b3-0e20-48c9-8843-cf574e833566"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:43.051422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051381    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051426    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-kube-api-access-mgl48\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051443    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f6de8b3-0e20-48c9-8843-cf574e833566-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051497    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051515    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051529    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051543    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051554    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.051653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.051567    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f6de8b3-0e20-48c9-8843-cf574e833566-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.065044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.065014    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:30:43.101243 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.101172    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:30:43.118069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.118030    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"]
Oct 31 00:30:43.143097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.143058    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"]
Oct 31 00:30:43.154456 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.154430    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng\" (UniqueName: \"kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng\") pod \"651895ba-2216-4409-9a0a-3c190aed5f52\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") "
Oct 31 00:30:43.154705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.154689    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume\") pod \"34836a0f-c53e-4dee-a76b-85a954663e7a\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") "
Oct 31 00:30:43.154768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.154731    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume\") pod \"651895ba-2216-4409-9a0a-3c190aed5f52\" (UID: \"651895ba-2216-4409-9a0a-3c190aed5f52\") "
Oct 31 00:30:43.154894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.154857    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf\" (UniqueName: \"kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf\") pod \"34836a0f-c53e-4dee-a76b-85a954663e7a\" (UID: \"34836a0f-c53e-4dee-a76b-85a954663e7a\") "
Oct 31 00:30:43.155334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.155307    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume" pod "651895ba-2216-4409-9a0a-3c190aed5f52" (UID: "651895ba-2216-4409-9a0a-3c190aed5f52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:43.155587 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.155563    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume" pod "34836a0f-c53e-4dee-a76b-85a954663e7a" (UID: "34836a0f-c53e-4dee-a76b-85a954663e7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:43.156082 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.156064    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34836a0f-c53e-4dee-a76b-85a954663e7a-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.156233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.156209    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651895ba-2216-4409-9a0a-3c190aed5f52-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.157209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.157096    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng" pod "651895ba-2216-4409-9a0a-3c190aed5f52" (UID: "651895ba-2216-4409-9a0a-3c190aed5f52"). InnerVolumeSpecName "kube-api-access-d5mng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:43.167605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.164868    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf" pod "34836a0f-c53e-4dee-a76b-85a954663e7a" (UID: "34836a0f-c53e-4dee-a76b-85a954663e7a"). InnerVolumeSpecName "kube-api-access-pxlkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:43.257596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.257557    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5mng\" (UniqueName: \"kubernetes.io/projected/651895ba-2216-4409-9a0a-3c190aed5f52-kube-api-access-d5mng\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.257596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.257594    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlkf\" (UniqueName: \"kubernetes.io/projected/34836a0f-c53e-4dee-a76b-85a954663e7a-kube-api-access-pxlkf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:43.500984 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.500938    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"]
Oct 31 00:30:43.506348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.506308    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31108d-5m4pq"]
Oct 31 00:30:43.635756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.635718    5078 generic.go:356] "Generic (PLEG): container finished" podID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerID="c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7" exitCode=0
Oct 31 00:30:43.635970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.635806    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"
Oct 31 00:30:43.635970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.635810    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" event={"ID":"9f6de8b3-0e20-48c9-8843-cf574e833566","Type":"ContainerDied","Data":"c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7"}
Oct 31 00:30:43.635970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.635872    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9" event={"ID":"9f6de8b3-0e20-48c9-8843-cf574e833566","Type":"ContainerDied","Data":"d06aa93c3cab5f75eadde89472fcd75d92a8ce6c1db396b895382c07984e2a91"}
Oct 31 00:30:43.635970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.635899    5078 scope.go:122] "RemoveContainer" containerID="c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7"
Oct 31 00:30:43.637782 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.637761    5078 generic.go:356] "Generic (PLEG): container finished" podID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerID="3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d" exitCode=0
Oct 31 00:30:43.637908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.637832    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"
Oct 31 00:30:43.638364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.637789    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" event={"ID":"34836a0f-c53e-4dee-a76b-85a954663e7a","Type":"ContainerDied","Data":"3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d"}
Oct 31 00:30:43.638641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.638612    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw" event={"ID":"34836a0f-c53e-4dee-a76b-85a954663e7a","Type":"ContainerDied","Data":"3c039819dcce83d8fc3254ea24f0be5e759083c915528e909500a6b47660b055"}
Oct 31 00:30:43.640635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.640612    5078 generic.go:356] "Generic (PLEG): container finished" podID="651895ba-2216-4409-9a0a-3c190aed5f52" containerID="470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804" exitCode=0
Oct 31 00:30:43.641016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.640990    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" event={"ID":"651895ba-2216-4409-9a0a-3c190aed5f52","Type":"ContainerDied","Data":"470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804"}
Oct 31 00:30:43.641151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.641042    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g" event={"ID":"651895ba-2216-4409-9a0a-3c190aed5f52","Type":"ContainerDied","Data":"4b31f15220027b52fa6e092bca014cb967f570c194c000674327b3af329a352c"}
Oct 31 00:30:43.641151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.641109    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"
Oct 31 00:30:43.650352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.650338    5078 scope.go:122] "RemoveContainer" containerID="a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"
Oct 31 00:30:43.660915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.660901    5078 scope.go:122] "RemoveContainer" containerID="c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7"
Oct 31 00:30:43.661162 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:43.661143    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7\": container with ID starting with c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7 not found: ID does not exist" containerID="c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7"
Oct 31 00:30:43.661203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.661170    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7"} err="failed to get container status \"c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7\": rpc error: code = NotFound desc = could not find container \"c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7\": container with ID starting with c20f0325a07bf02a3b9d7edf9712196e7ca9da30755611e691bcb3b7ed44fac7 not found: ID does not exist"
Oct 31 00:30:43.661203 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.661186    5078 scope.go:122] "RemoveContainer" containerID="a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"
Oct 31 00:30:43.661415 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:43.661401    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb\": container with ID starting with a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb not found: ID does not exist" containerID="a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"
Oct 31 00:30:43.661455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.661422    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb"} err="failed to get container status \"a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb\": rpc error: code = NotFound desc = could not find container \"a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb\": container with ID starting with a422f005c42c8bbca07d4d12929394300b226a3d6a0a65ca8cf6d341df2402bb not found: ID does not exist"
Oct 31 00:30:43.661455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.661436    5078 scope.go:122] "RemoveContainer" containerID="3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d"
Oct 31 00:30:43.672149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.671965    5078 scope.go:122] "RemoveContainer" containerID="3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d"
Oct 31 00:30:43.672475 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:43.672325    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d\": container with ID starting with 3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d not found: ID does not exist" containerID="3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d"
Oct 31 00:30:43.672475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.672361    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d"} err="failed to get container status \"3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d\": rpc error: code = NotFound desc = could not find container \"3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d\": container with ID starting with 3a7b175ee680a3e1c1eac66a2287d9c1e2c3b23259ed31114748ba4bfdcf427d not found: ID does not exist"
Oct 31 00:30:43.672475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.672384    5078 scope.go:122] "RemoveContainer" containerID="470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804"
Oct 31 00:30:43.675039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.675016    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"]
Oct 31 00:30:43.679126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.679105    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c60826-bbfdc6d6c-fls8g"]
Oct 31 00:30:43.683683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.683667    5078 scope.go:122] "RemoveContainer" containerID="470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804"
Oct 31 00:30:43.683930 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:43.683912    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804\": container with ID starting with 470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804 not found: ID does not exist" containerID="470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804"
Oct 31 00:30:43.683983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.683938    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804"} err="failed to get container status \"470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804\": rpc error: code = NotFound desc = could not find container \"470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804\": container with ID starting with 470e48ba1fb4e42719ff568a08d02107333498f974d99b512509bec9816ec804 not found: ID does not exist"
Oct 31 00:30:43.691321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.691298    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"]
Oct 31 00:30:43.696474 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.696444    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c60826-7c96fd764f-xd9dw"]
Oct 31 00:30:43.714536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.714515    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"]
Oct 31 00:30:43.718133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:43.718111    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0d4ad9-openshift-default-5898445b79-2cbs9"]
Oct 31 00:30:44.195523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:44.195486    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" path="/var/lib/kubelet/pods/34836a0f-c53e-4dee-a76b-85a954663e7a/volumes"
Oct 31 00:30:44.195963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:44.195947    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" path="/var/lib/kubelet/pods/651895ba-2216-4409-9a0a-3c190aed5f52/volumes"
Oct 31 00:30:44.196401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:44.196387    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" path="/var/lib/kubelet/pods/9f6de8b3-0e20-48c9-8843-cf574e833566/volumes"
Oct 31 00:30:44.197168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:44.197151    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ad2eba-5caa-4231-87d5-50431b292d29" path="/var/lib/kubelet/pods/f4ad2eba-5caa-4231-87d5-50431b292d29/volumes"
Oct 31 00:30:45.549020 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:45.548987    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"]
Oct 31 00:30:45.552779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:45.552754    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c60826-nftv6"]
Oct 31 00:30:46.196764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:46.196727    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1218a7-d5bd-42e7-99d0-281a1090b882" path="/var/lib/kubelet/pods/0d1218a7-d5bd-42e7-99d0-281a1090b882/volumes"
Oct 31 00:30:48.505750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:48.505710    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"]
Oct 31 00:30:50.828762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.828722    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"]
Oct 31 00:30:50.829609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829589    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9d904c7e-e1c8-41fa-9397-3691813d172e" containerName="revoke-keys"
Oct 31 00:30:50.829609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829608    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d904c7e-e1c8-41fa-9397-3691813d172e" containerName="revoke-keys"
Oct 31 00:30:50.829754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829624    5078 cpu_manager.go:414] "containerMap: removing container" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing"
Oct 31 00:30:50.829754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829629    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing"
Oct 31 00:30:50.829754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829640    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy"
Oct 31 00:30:50.829754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829645    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy"
Oct 31 00:30:50.829754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829651    5078 cpu_manager.go:414] "containerMap: removing container" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing"
Oct 31 00:30:50.829754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829656    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing"
Oct 31 00:30:50.829952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829766    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy"
Oct 31 00:30:50.829952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829781    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d904c7e-e1c8-41fa-9397-3691813d172e" containerName="revoke-keys"
Oct 31 00:30:50.829952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829789    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="651895ba-2216-4409-9a0a-3c190aed5f52" containerName="payload-pre-processing"
Oct 31 00:30:50.829952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829795    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy"
Oct 31 00:30:50.829952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.829800    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="34836a0f-c53e-4dee-a76b-85a954663e7a" containerName="payload-processing"
Oct 31 00:30:50.834472 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.834439    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:50.846920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.845442    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"]
Oct 31 00:30:50.906431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.906398    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:50.906612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.906529    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:30:50.967000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.966965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:50.967167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.967086    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:50.967269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:50.967253    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk59n\" (UniqueName: \"kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.068519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.068477    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.068693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.068586    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.068788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.068772    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk59n\" (UniqueName: \"kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.069299 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.069222    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.071541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.071520    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.092884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.092831    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk59n\" (UniqueName: \"kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.155249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.155219    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:51.456481 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.456437    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"]
Oct 31 00:30:51.462042 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:51.462005    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dd39d3_7570_4dbf_88c1_7b72a5e28e5c.slice/crio-afbd39f0b7ebc7f2665387a47553935e7e0be3812cb1a2e6a54258b951e5c92b WatchSource:0}: Error finding container afbd39f0b7ebc7f2665387a47553935e7e0be3812cb1a2e6a54258b951e5c92b: Status 404 returned error can't find the container with id afbd39f0b7ebc7f2665387a47553935e7e0be3812cb1a2e6a54258b951e5c92b
Oct 31 00:30:51.714493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:51.714373    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf" event={"ID":"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c","Type":"ContainerStarted","Data":"afbd39f0b7ebc7f2665387a47553935e7e0be3812cb1a2e6a54258b951e5c92b"}
Oct 31 00:30:52.724359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:52.724328    5078 generic.go:356] "Generic (PLEG): container finished" podID="33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" containerID="6282c41fcdc477da840df51d309ab2b1dbf11181ad200a2b6ee4393df3e1a303" exitCode=0
Oct 31 00:30:52.724713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:52.724402    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf" event={"ID":"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c","Type":"ContainerDied","Data":"6282c41fcdc477da840df51d309ab2b1dbf11181ad200a2b6ee4393df3e1a303"}
Oct 31 00:30:53.507351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.507292    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" containerID="cri-o://503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5" gracePeriod=25
Oct 31 00:30:53.683233 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.683196    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"]
Oct 31 00:30:53.683869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.683853    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy"
Oct 31 00:30:53.683921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.683871    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6de8b3-0e20-48c9-8843-cf574e833566" containerName="istio-proxy"
Oct 31 00:30:53.687773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.687750    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.709730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.702054    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"]
Oct 31 00:30:53.829631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.829589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.830759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.830688    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkstq\" (UniqueName: \"kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.831214 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.831186    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.935625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.932425    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.935625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.932551    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.935625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.932585    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkstq\" (UniqueName: \"kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.935625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.933982    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.935987 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.935814    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.947710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.947681    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkstq\" (UniqueName: \"kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq\") pod \"maas-api-revoke-sub-03e6697a-xn5lm\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:53.984170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:53.984142    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:54.010190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.010161    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:54.033682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.033653    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n\" (UniqueName: \"kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n\") pod \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") "
Oct 31 00:30:54.033975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.033837    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca\") pod \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") "
Oct 31 00:30:54.033975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.033909    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca\") pod \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\" (UID: \"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c\") "
Oct 31 00:30:54.035156 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.035042    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca" pod "33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" (UID: "33dd39d3-7570-4dbf-88c1-7b72a5e28e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:54.037080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.037045    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:54.041026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.039326    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n" pod "33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" (UID: "33dd39d3-7570-4dbf-88c1-7b72a5e28e5c"). InnerVolumeSpecName "kube-api-access-pk59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:54.041026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.040029    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca" pod "33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" (UID: "33dd39d3-7570-4dbf-88c1-7b72a5e28e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:54.139125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.139088    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:54.139263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.139134    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk59n\" (UniqueName: \"kubernetes.io/projected/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c-kube-api-access-pk59n\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:54.369964 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:54.369932    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0972488_0d0d_482e_8bda_9fa193fbf1ef.slice/crio-7d364ddc5b016c004ac2505ba1570473b263b3efbbe933d34ebf189a9a12a2f6 WatchSource:0}: Error finding container 7d364ddc5b016c004ac2505ba1570473b263b3efbbe933d34ebf189a9a12a2f6: Status 404 returned error can't find the container with id 7d364ddc5b016c004ac2505ba1570473b263b3efbbe933d34ebf189a9a12a2f6
Oct 31 00:30:54.370115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.369978    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"]
Oct 31 00:30:54.373489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.373452    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"]
Oct 31 00:30:54.374076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.374061    5078 cpu_manager.go:414] "containerMap: removing container" podUID="33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" containerName="revoke-keys"
Oct 31 00:30:54.374148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.374080    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" containerName="revoke-keys"
Oct 31 00:30:54.374247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.374233    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" containerName="revoke-keys"
Oct 31 00:30:54.399591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.399563    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.402759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.402735    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"]
Oct 31 00:30:54.543988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.543947    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2thh\" (UniqueName: \"kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.544174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.544130    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.544260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.544178    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.645954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.645918    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.646134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.646002    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.646239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.646220    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2thh\" (UniqueName: \"kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.646694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.646669    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.648654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.648631    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.659297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.659272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2thh\" (UniqueName: \"kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.729654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.729623    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:54.763639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.763597    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm" event={"ID":"f0972488-0d0d-482e-8bda-9fa193fbf1ef","Type":"ContainerStarted","Data":"7d364ddc5b016c004ac2505ba1570473b263b3efbbe933d34ebf189a9a12a2f6"}
Oct 31 00:30:54.765817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.765790    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf" event={"ID":"33dd39d3-7570-4dbf-88c1-7b72a5e28e5c","Type":"ContainerDied","Data":"afbd39f0b7ebc7f2665387a47553935e7e0be3812cb1a2e6a54258b951e5c92b"}
Oct 31 00:30:54.765948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.765840    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbd39f0b7ebc7f2665387a47553935e7e0be3812cb1a2e6a54258b951e5c92b"
Oct 31 00:30:54.765948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.765907    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"
Oct 31 00:30:54.835887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:54.835850    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:55.020401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:55.020374    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"]
Oct 31 00:30:55.021188 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:30:55.021160    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22b2119_a2e7_41d3_802f_64203d2ca060.slice/crio-daa8fe52fb20baa5653dc42363e1e35bd3a6205e5974841284221c22974aff86 WatchSource:0}: Error finding container daa8fe52fb20baa5653dc42363e1e35bd3a6205e5974841284221c22974aff86: Status 404 returned error can't find the container with id daa8fe52fb20baa5653dc42363e1e35bd3a6205e5974841284221c22974aff86
Oct 31 00:30:55.775750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:55.775718    5078 generic.go:356] "Generic (PLEG): container finished" podID="f0972488-0d0d-482e-8bda-9fa193fbf1ef" containerID="23488c43f37ac5ab81a1f1a312e38a2c54a22751fc3e67d974a2746e179188b8" exitCode=0
Oct 31 00:30:55.776006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:55.775805    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm" event={"ID":"f0972488-0d0d-482e-8bda-9fa193fbf1ef","Type":"ContainerDied","Data":"23488c43f37ac5ab81a1f1a312e38a2c54a22751fc3e67d974a2746e179188b8"}
Oct 31 00:30:55.777398 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:55.777374    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7" event={"ID":"a22b2119-a2e7-41d3-802f-64203d2ca060","Type":"ContainerStarted","Data":"0615da6a5189950fcbcdbef0db8d5e72a1f539e6c352335a9f204e306be13b27"}
Oct 31 00:30:55.777521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:55.777412    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7" event={"ID":"a22b2119-a2e7-41d3-802f-64203d2ca060","Type":"ContainerStarted","Data":"daa8fe52fb20baa5653dc42363e1e35bd3a6205e5974841284221c22974aff86"}
Oct 31 00:30:55.816231 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:55.816149    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7" podStartSLOduration=1.8161365489999999 podStartE2EDuration="1.816136549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:55.815060217 +0000 UTC m=+2223.761430935" watchObservedRunningTime="2026-10-31 00:30:55.816136549 +0000 UTC m=+2223.762507297"
Oct 31 00:30:56.788372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:56.788333    5078 generic.go:356] "Generic (PLEG): container finished" podID="a22b2119-a2e7-41d3-802f-64203d2ca060" containerID="0615da6a5189950fcbcdbef0db8d5e72a1f539e6c352335a9f204e306be13b27" exitCode=0
Oct 31 00:30:56.788902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:56.788419    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7" event={"ID":"a22b2119-a2e7-41d3-802f-64203d2ca060","Type":"ContainerDied","Data":"0615da6a5189950fcbcdbef0db8d5e72a1f539e6c352335a9f204e306be13b27"}
Oct 31 00:30:56.934483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:56.933958    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"]
Oct 31 00:30:56.934483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:56.934221    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy" containerID="cri-o://c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0" gracePeriod=30
Oct 31 00:30:57.017059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.017027    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:57.089564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.081967    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca\") pod \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") "
Oct 31 00:30:57.089564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.082050    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca\") pod \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") "
Oct 31 00:30:57.089564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.082257    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq\" (UniqueName: \"kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq\") pod \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\" (UID: \"f0972488-0d0d-482e-8bda-9fa193fbf1ef\") "
Oct 31 00:30:57.089564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.084882    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca" pod "f0972488-0d0d-482e-8bda-9fa193fbf1ef" (UID: "f0972488-0d0d-482e-8bda-9fa193fbf1ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:57.104493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.100886    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:57.126778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.126184    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq" pod "f0972488-0d0d-482e-8bda-9fa193fbf1ef" (UID: "f0972488-0d0d-482e-8bda-9fa193fbf1ef"). InnerVolumeSpecName "kube-api-access-qkstq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:57.126778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.126401    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca" pod "f0972488-0d0d-482e-8bda-9fa193fbf1ef" (UID: "f0972488-0d0d-482e-8bda-9fa193fbf1ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:57.202277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.202242    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0972488-0d0d-482e-8bda-9fa193fbf1ef-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:57.202277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.202269    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkstq\" (UniqueName: \"kubernetes.io/projected/f0972488-0d0d-482e-8bda-9fa193fbf1ef-kube-api-access-qkstq\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:57.797735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.797703    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"
Oct 31 00:30:57.798147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.797744    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm" event={"ID":"f0972488-0d0d-482e-8bda-9fa193fbf1ef","Type":"ContainerDied","Data":"7d364ddc5b016c004ac2505ba1570473b263b3efbbe933d34ebf189a9a12a2f6"}
Oct 31 00:30:57.798147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.797795    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d364ddc5b016c004ac2505ba1570473b263b3efbbe933d34ebf189a9a12a2f6"
Oct 31 00:30:57.967192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:57.967162    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:58.115802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.115720    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca\") pod \"a22b2119-a2e7-41d3-802f-64203d2ca060\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") "
Oct 31 00:30:58.115961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.115828    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca\") pod \"a22b2119-a2e7-41d3-802f-64203d2ca060\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") "
Oct 31 00:30:58.115961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.115921    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh\" (UniqueName: \"kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh\") pod \"a22b2119-a2e7-41d3-802f-64203d2ca060\" (UID: \"a22b2119-a2e7-41d3-802f-64203d2ca060\") "
Oct 31 00:30:58.116225 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.116190    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca" pod "a22b2119-a2e7-41d3-802f-64203d2ca060" (UID: "a22b2119-a2e7-41d3-802f-64203d2ca060"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:58.118144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.118112    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca" pod "a22b2119-a2e7-41d3-802f-64203d2ca060" (UID: "a22b2119-a2e7-41d3-802f-64203d2ca060"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:58.118239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.118220    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh" pod "a22b2119-a2e7-41d3-802f-64203d2ca060" (UID: "a22b2119-a2e7-41d3-802f-64203d2ca060"). InnerVolumeSpecName "kube-api-access-l2thh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:58.146377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.146344    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main" containerID="cri-o://80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6" gracePeriod=45
Oct 31 00:30:58.167641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.167608    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" containerID="cri-o://43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71" gracePeriod=45
Oct 31 00:30:58.217268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.217243    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2thh\" (UniqueName: \"kubernetes.io/projected/a22b2119-a2e7-41d3-802f-64203d2ca060-kube-api-access-l2thh\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.217380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.217270    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.217380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.217285    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22b2119-a2e7-41d3-802f-64203d2ca060-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.353620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.351887    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 00:30:58.496309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.496272    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:30:58.521761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.521734    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:30:58.624607 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624524    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb\" (UniqueName: \"kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb\") pod \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") "
Oct 31 00:30:58.624840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624649    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home\") pod \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") "
Oct 31 00:30:58.624840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624688    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs\") pod \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") "
Oct 31 00:30:58.624840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624733    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir\") pod \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") "
Oct 31 00:30:58.624840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624788    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache\") pod \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") "
Oct 31 00:30:58.625083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624900    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g\" (UniqueName: \"kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g\") pod \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") "
Oct 31 00:30:58.625083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.624943    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs\") pod \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") "
Oct 31 00:30:58.625083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625001    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm\") pod \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") "
Oct 31 00:30:58.625083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625041    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm\") pod \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") "
Oct 31 00:30:58.625083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625079    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home\") pod \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\" (UID: \"b0dc259b-ccb1-4912-9dc8-c52852f55e75\") "
Oct 31 00:30:58.625359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir\") pod \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") "
Oct 31 00:30:58.625359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625225    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache\") pod \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\" (UID: \"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd\") "
Oct 31 00:30:58.625719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625693    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home" pod "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" (UID: "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.625822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625793    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home" pod "b0dc259b-ccb1-4912-9dc8-c52852f55e75" (UID: "b0dc259b-ccb1-4912-9dc8-c52852f55e75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.625940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625919    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache" pod "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" (UID: "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.626001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625936    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.626001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.625955    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.626159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.626134    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir" pod "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" (UID: "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.626381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.626360    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache" pod "b0dc259b-ccb1-4912-9dc8-c52852f55e75" (UID: "b0dc259b-ccb1-4912-9dc8-c52852f55e75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.626506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.626482    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir" pod "b0dc259b-ccb1-4912-9dc8-c52852f55e75" (UID: "b0dc259b-ccb1-4912-9dc8-c52852f55e75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.629773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.629616    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs" pod "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" (UID: "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:58.629773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.629701    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g" pod "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" (UID: "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd"). InnerVolumeSpecName "kube-api-access-szx2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:58.629773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.629707    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm" pod "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" (UID: "6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.630016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.629950    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm" pod "b0dc259b-ccb1-4912-9dc8-c52852f55e75" (UID: "b0dc259b-ccb1-4912-9dc8-c52852f55e75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:58.630191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.630166    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb" pod "b0dc259b-ccb1-4912-9dc8-c52852f55e75" (UID: "b0dc259b-ccb1-4912-9dc8-c52852f55e75"). InnerVolumeSpecName "kube-api-access-828gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:58.630844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.630818    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs" pod "b0dc259b-ccb1-4912-9dc8-c52852f55e75" (UID: "b0dc259b-ccb1-4912-9dc8-c52852f55e75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:58.727878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727846    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.727878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727875    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727887    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828gb\" (UniqueName: \"kubernetes.io/projected/b0dc259b-ccb1-4912-9dc8-c52852f55e75-kube-api-access-828gb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727896    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727906    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727913    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727922    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szx2g\" (UniqueName: \"kubernetes.io/projected/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-kube-api-access-szx2g\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727930    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0dc259b-ccb1-4912-9dc8-c52852f55e75-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727938    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.728123 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.727946    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0dc259b-ccb1-4912-9dc8-c52852f55e75-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:58.808208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.808178    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"
Oct 31 00:30:58.808208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.808197    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7" event={"ID":"a22b2119-a2e7-41d3-802f-64203d2ca060","Type":"ContainerDied","Data":"daa8fe52fb20baa5653dc42363e1e35bd3a6205e5974841284221c22974aff86"}
Oct 31 00:30:58.808781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.808250    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daa8fe52fb20baa5653dc42363e1e35bd3a6205e5974841284221c22974aff86"
Oct 31 00:30:58.810160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.810130    5078 generic.go:356] "Generic (PLEG): container finished" podID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerID="80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6" exitCode=0
Oct 31 00:30:58.810308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.810180    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"
Oct 31 00:30:58.810308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.810229    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" event={"ID":"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd","Type":"ContainerDied","Data":"80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6"}
Oct 31 00:30:58.810308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.810267    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd" event={"ID":"6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd","Type":"ContainerDied","Data":"1a026f967063b42c914cb7de419c673c15873021374222e2b22eb910a14554ce"}
Oct 31 00:30:58.810308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.810286    5078 scope.go:122] "RemoveContainer" containerID="80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6"
Oct 31 00:30:58.813339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.813314    5078 generic.go:356] "Generic (PLEG): container finished" podID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerID="43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71" exitCode=0
Oct 31 00:30:58.813557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.813370    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"
Oct 31 00:30:58.813557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.813413    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" event={"ID":"b0dc259b-ccb1-4912-9dc8-c52852f55e75","Type":"ContainerDied","Data":"43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71"}
Oct 31 00:30:58.813557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.813491    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4" event={"ID":"b0dc259b-ccb1-4912-9dc8-c52852f55e75","Type":"ContainerDied","Data":"9e00c984f3fcc884cf0761f409cc629e3b622b0d80128b5cd9e8d0468ef01124"}
Oct 31 00:30:58.824709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.824690    5078 scope.go:122] "RemoveContainer" containerID="80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6"
Oct 31 00:30:58.825028 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:58.825006    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6\": container with ID starting with 80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6 not found: ID does not exist" containerID="80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6"
Oct 31 00:30:58.825129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.825035    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6"} err="failed to get container status \"80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6\": rpc error: code = NotFound desc = could not find container \"80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6\": container with ID starting with 80091ef78ddb8e3d30fa59afad6bbbab7d05d79c533aa7d2c042333b8d048af6 not found: ID does not exist"
Oct 31 00:30:58.825129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.825056    5078 scope.go:122] "RemoveContainer" containerID="43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71"
Oct 31 00:30:58.836426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.836326    5078 scope.go:122] "RemoveContainer" containerID="43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71"
Oct 31 00:30:58.836743 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:30:58.836714    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71\": container with ID starting with 43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71 not found: ID does not exist" containerID="43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71"
Oct 31 00:30:58.836882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.836751    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71"} err="failed to get container status \"43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71\": rpc error: code = NotFound desc = could not find container \"43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71\": container with ID starting with 43fdd636b69834529e31d566a634b4668ffad12b385d3493803a4ac6f22d2d71 not found: ID does not exist"
Oct 31 00:30:58.848399 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.848376    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"]
Oct 31 00:30:58.851488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.851454    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-2-w6-0402df-kserve-7fbd4ddbb4-7qwhd"]
Oct 31 00:30:58.866703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.866677    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"]
Oct 31 00:30:58.869828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:30:58.869806    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/simulator-w6-0402df-kserve-84cd889645-8tqm4"]
Oct 31 00:31:00.198713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:00.198679    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" path="/var/lib/kubelet/pods/6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd/volumes"
Oct 31 00:31:00.199352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:00.199334    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" path="/var/lib/kubelet/pods/b0dc259b-ccb1-4912-9dc8-c52852f55e75/volumes"
Oct 31 00:31:01.006054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:01.005999    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"]
Oct 31 00:31:01.634804 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:01.634771    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"]
Oct 31 00:31:01.647969 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:01.647938    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-14e39b-mg8bf"]
Oct 31 00:31:02.199135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.199103    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33dd39d3-7570-4dbf-88c1-7b72a5e28e5c" path="/var/lib/kubelet/pods/33dd39d3-7570-4dbf-88c1-7b72a5e28e5c/volumes"
Oct 31 00:31:02.241881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.241858    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:31:02.281381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.281349    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.281646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.281625    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.281824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.281809    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.282039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.281974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.282215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.281877    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:02.282287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.282183    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:02.283519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.282201    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.285628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.284106    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43cd4be-8675-4420-af28-f91e8e0d04b5-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.285628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.284131    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.298534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.298508    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:02.298658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.298569    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:02.305121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.305096    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:31:02.384976 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.384943    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.385195 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.384989    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.385195 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385030    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.385348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385313    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs\") pod \"b43cd4be-8675-4420-af28-f91e8e0d04b5\" (UID: \"b43cd4be-8675-4420-af28-f91e8e0d04b5\") "
Oct 31 00:31:02.385571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385535    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:02.385709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385628    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:02.385954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385935    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.386034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385958    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.386034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385968    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.386034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385976    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.386034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.385985    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.387583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.387557    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:02.388158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.387842    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp" pod "b43cd4be-8675-4420-af28-f91e8e0d04b5" (UID: "b43cd4be-8675-4420-af28-f91e8e0d04b5"). InnerVolumeSpecName "kube-api-access-m7kzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:02.491626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.490382    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kzp\" (UniqueName: \"kubernetes.io/projected/b43cd4be-8675-4420-af28-f91e8e0d04b5-kube-api-access-m7kzp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.491626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.490420    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43cd4be-8675-4420-af28-f91e8e0d04b5-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:02.848043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.847958    5078 generic.go:356] "Generic (PLEG): container finished" podID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerID="c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0" exitCode=0
Oct 31 00:31:02.848043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.848010    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"
Oct 31 00:31:02.848546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.848015    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" event={"ID":"b43cd4be-8675-4420-af28-f91e8e0d04b5","Type":"ContainerDied","Data":"c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0"}
Oct 31 00:31:02.848546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.848129    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24" event={"ID":"b43cd4be-8675-4420-af28-f91e8e0d04b5","Type":"ContainerDied","Data":"ea7ba27ea4f73ddff42309ef67f2a2dc487690a13c0c6a56b1a7b395656b4470"}
Oct 31 00:31:02.848546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.848152    5078 scope.go:122] "RemoveContainer" containerID="c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0"
Oct 31 00:31:02.859479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.859445    5078 scope.go:122] "RemoveContainer" containerID="7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"
Oct 31 00:31:02.869659 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.869631    5078 scope.go:122] "RemoveContainer" containerID="c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0"
Oct 31 00:31:02.869946 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:02.869928    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0\": container with ID starting with c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0 not found: ID does not exist" containerID="c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0"
Oct 31 00:31:02.870039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.869957    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0"} err="failed to get container status \"c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0\": rpc error: code = NotFound desc = could not find container \"c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0\": container with ID starting with c679553d3d7d8925e2d63cdb0fa469ebe231bdba3a1cf04838a65260c7ead9c0 not found: ID does not exist"
Oct 31 00:31:02.870039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.869972    5078 scope.go:122] "RemoveContainer" containerID="7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"
Oct 31 00:31:02.870240 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:02.870220    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a\": container with ID starting with 7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a not found: ID does not exist" containerID="7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"
Oct 31 00:31:02.870296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.870244    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a"} err="failed to get container status \"7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a\": rpc error: code = NotFound desc = could not find container \"7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a\": container with ID starting with 7926256521bb9b4c4ea6858fdaf9cdc45469ca029f3a3c096798e73998b73b1a not found: ID does not exist"
Oct 31 00:31:02.878642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.878616    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"]
Oct 31 00:31:02.883632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:02.883608    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-6hx24"]
Oct 31 00:31:04.201815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:04.201772    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" path="/var/lib/kubelet/pods/b43cd4be-8675-4420-af28-f91e8e0d04b5/volumes"
Oct 31 00:31:04.714014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:04.713970    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"]
Oct 31 00:31:04.728422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:04.728385    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"]
Oct 31 00:31:04.836113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:04.836072    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:05.919107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:05.919075    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"]
Oct 31 00:31:05.921165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:05.921140    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0402df-jr58c"]
Oct 31 00:31:06.006755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.006711    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing" containerID="cri-o://90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c" gracePeriod=25
Oct 31 00:31:06.203974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.203881    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d904c7e-e1c8-41fa-9397-3691813d172e" path="/var/lib/kubelet/pods/9d904c7e-e1c8-41fa-9397-3691813d172e/volumes"
Oct 31 00:31:06.346370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.346343    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:31:06.426723 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:06.426682    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:31:06.466366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.466292    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume\") pod \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") "
Oct 31 00:31:06.466530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.466423    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48\" (UniqueName: \"kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48\") pod \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\" (UID: \"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c\") "
Oct 31 00:31:06.466681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.466660    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume" pod "2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" (UID: "2451ffa2-2e2c-4f9e-a731-e6969efc1e4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:06.466845 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.466831    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:06.468687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.468653    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48" pod "2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" (UID: "2451ffa2-2e2c-4f9e-a731-e6969efc1e4c"). InnerVolumeSpecName "kube-api-access-98j48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:06.567688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.567658    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98j48\" (UniqueName: \"kubernetes.io/projected/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c-kube-api-access-98j48\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:06.910166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.910133    5078 generic.go:356] "Generic (PLEG): container finished" podID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerID="90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c" exitCode=0
Oct 31 00:31:06.910530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.910384    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" event={"ID":"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c","Type":"ContainerDied","Data":"90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c"}
Oct 31 00:31:06.910686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.910401    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"
Oct 31 00:31:06.911175 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.910969    5078 scope.go:122] "RemoveContainer" containerID="90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c"
Oct 31 00:31:06.919749 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.919720    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt" event={"ID":"2451ffa2-2e2c-4f9e-a731-e6969efc1e4c","Type":"ContainerDied","Data":"b776ed84452a76fbbde7c7db845fa57f187fd8e63983357b032b58a55add1983"}
Oct 31 00:31:06.931427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.931407    5078 scope.go:122] "RemoveContainer" containerID="90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c"
Oct 31 00:31:06.932017 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:06.931996    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c\": container with ID starting with 90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c not found: ID does not exist" containerID="90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c"
Oct 31 00:31:06.932097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.932025    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c"} err="failed to get container status \"90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c\": rpc error: code = NotFound desc = could not find container \"90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c\": container with ID starting with 90e5e3018096c5860a9dc09ee5674f7d0f52a66370ec1c86b3e130565bb83e5c not found: ID does not exist"
Oct 31 00:31:06.955963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.955929    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"]
Oct 31 00:31:06.993489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:06.970272    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-14e39b-7cbdff884c-lq5zt"]
Oct 31 00:31:07.408075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:07.408007    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"]
Oct 31 00:31:07.410624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:07.410589    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-945abd-f62t7"]
Oct 31 00:31:08.208262 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:08.208226    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" path="/var/lib/kubelet/pods/2451ffa2-2e2c-4f9e-a731-e6969efc1e4c/volumes"
Oct 31 00:31:08.208682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:08.208651    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22b2119-a2e7-41d3-802f-64203d2ca060" path="/var/lib/kubelet/pods/a22b2119-a2e7-41d3-802f-64203d2ca060/volumes"
Oct 31 00:31:09.730576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:09.730530    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr" podUID="f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" containerName="payload-pre-processing" containerID="cri-o://4d3324909ec1f4b91c968d4cb4733abf056a8c5604279a6401905818852f38db" gracePeriod=25
Oct 31 00:31:09.866549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:09.866502    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp" podUID="1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" containerName="payload-processing" containerID="cri-o://45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a" gracePeriod=25
Oct 31 00:31:09.940039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:09.940005    5078 generic.go:356] "Generic (PLEG): container finished" podID="f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" containerID="4d3324909ec1f4b91c968d4cb4733abf056a8c5604279a6401905818852f38db" exitCode=0
Oct 31 00:31:09.940192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:09.940177    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr" event={"ID":"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d","Type":"ContainerDied","Data":"4d3324909ec1f4b91c968d4cb4733abf056a8c5604279a6401905818852f38db"}
Oct 31 00:31:10.131090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.131058    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:31:10.246612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.242314    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48\" (UniqueName: \"kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48\") pod \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") "
Oct 31 00:31:10.246612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.242445    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume\") pod \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\" (UID: \"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d\") "
Oct 31 00:31:10.246612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.243363    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume" pod "f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" (UID: "f5b6ab70-6e9a-49af-ac30-8aebecc4b64d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:10.255049 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.255014    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48" pod "f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" (UID: "f5b6ab70-6e9a-49af-ac30-8aebecc4b64d"). InnerVolumeSpecName "kube-api-access-zmm48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:10.324487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.324435    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:31:10.344355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.344327    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:10.344355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.344351    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmm48\" (UniqueName: \"kubernetes.io/projected/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d-kube-api-access-zmm48\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:10.444888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.444856    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume\") pod \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") "
Oct 31 00:31:10.444888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.444893    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh\" (UniqueName: \"kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh\") pod \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\" (UID: \"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6\") "
Oct 31 00:31:10.445828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.445797    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume" pod "1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" (UID: "1fe746c1-6f69-4a40-b7d2-2c351b8b65a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:10.447509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.447486    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh" pod "1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" (UID: "1fe746c1-6f69-4a40-b7d2-2c351b8b65a6"). InnerVolumeSpecName "kube-api-access-qttbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:10.546427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.546392    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qttbh\" (UniqueName: \"kubernetes.io/projected/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-kube-api-access-qttbh\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:10.546427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.546427    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:10.951785 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.951747    5078 generic.go:356] "Generic (PLEG): container finished" podID="1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" containerID="45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a" exitCode=0
Oct 31 00:31:10.952239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.951812    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"
Oct 31 00:31:10.952239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.951834    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp" event={"ID":"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6","Type":"ContainerDied","Data":"45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a"}
Oct 31 00:31:10.952239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.951886    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp" event={"ID":"1fe746c1-6f69-4a40-b7d2-2c351b8b65a6","Type":"ContainerDied","Data":"d235728b3fae84b330898aed00533874966ceba7aea5c4372ead4ed11c6180a4"}
Oct 31 00:31:10.952239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.951913    5078 scope.go:122] "RemoveContainer" containerID="45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a"
Oct 31 00:31:10.954650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.954613    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr" event={"ID":"f5b6ab70-6e9a-49af-ac30-8aebecc4b64d","Type":"ContainerDied","Data":"4331a750e8a5b4ae2effb84d8be46ccf84976355790f67a131712cd7cbe26540"}
Oct 31 00:31:10.954788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.954712    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"
Oct 31 00:31:10.985824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.985803    5078 scope.go:122] "RemoveContainer" containerID="45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a"
Oct 31 00:31:10.986786 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:10.986226    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a\": container with ID starting with 45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a not found: ID does not exist" containerID="45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a"
Oct 31 00:31:10.986786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.986269    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a"} err="failed to get container status \"45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a\": rpc error: code = NotFound desc = could not find container \"45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a\": container with ID starting with 45198258561abd269660d2a617d987dde34fe514682392631d76a837fb9ad74a not found: ID does not exist"
Oct 31 00:31:10.986786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:10.986297    5078 scope.go:122] "RemoveContainer" containerID="4d3324909ec1f4b91c968d4cb4733abf056a8c5604279a6401905818852f38db"
Oct 31 00:31:11.030267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:11.027437    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"]
Oct 31 00:31:11.065495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:11.058545    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-945abd-798696568f-rh7hp"]
Oct 31 00:31:11.065495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:11.064512    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"]
Oct 31 00:31:11.070485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:11.066722    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-945abd-84ffff5d5f-r8qcr"]
Oct 31 00:31:12.197975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.197938    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" path="/var/lib/kubelet/pods/1fe746c1-6f69-4a40-b7d2-2c351b8b65a6/volumes"
Oct 31 00:31:12.198387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.198362    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" path="/var/lib/kubelet/pods/f5b6ab70-6e9a-49af-ac30-8aebecc4b64d/volumes"
Oct 31 00:31:12.775059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775025    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"]
Oct 31 00:31:12.775642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775624    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" containerName="payload-processing"
Oct 31 00:31:12.775642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775642    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" containerName="payload-processing"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775656    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" containerName="payload-pre-processing"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775661    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" containerName="payload-pre-processing"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775674    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775679    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775691    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a22b2119-a2e7-41d3-802f-64203d2ca060" containerName="revoke-keys"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775696    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22b2119-a2e7-41d3-802f-64203d2ca060" containerName="revoke-keys"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775707    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775712    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775724    5078 cpu_manager.go:414] "containerMap: removing container" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775728    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775736    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775740    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main"
Oct 31 00:31:12.775741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775746    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775751    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775761    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f0972488-0d0d-482e-8bda-9fa193fbf1ef" containerName="revoke-keys"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775765    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0972488-0d0d-482e-8bda-9fa193fbf1ef" containerName="revoke-keys"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775840    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22b2119-a2e7-41d3-802f-64203d2ca060" containerName="revoke-keys"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775849    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef0b0ba-12f4-4405-89ad-fcc33ffa23fd" containerName="main"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775856    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="2451ffa2-2e2c-4f9e-a731-e6969efc1e4c" containerName="payload-processing"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775861    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775865    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0dc259b-ccb1-4912-9dc8-c52852f55e75" containerName="main"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775871    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0972488-0d0d-482e-8bda-9fa193fbf1ef" containerName="revoke-keys"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775881    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe746c1-6f69-4a40-b7d2-2c351b8b65a6" containerName="payload-processing"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775886    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b6ab70-6e9a-49af-ac30-8aebecc4b64d" containerName="payload-pre-processing"
Oct 31 00:31:12.776255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.775891    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43cd4be-8675-4420-af28-f91e8e0d04b5" containerName="istio-proxy"
Oct 31 00:31:12.780321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.780294    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.807287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.807229    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:31:12.807528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.807242    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:31:12.812033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.811999    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"]
Oct 31 00:31:12.882208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.882170    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.882390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.882325    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.882390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.882375    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2xp\" (UniqueName: \"kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.983818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.983773    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.984025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.983899    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.984025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.983929    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2xp\" (UniqueName: \"kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.984650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.984618    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:12.986864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:12.986842    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:13.001824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.001755    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2xp\" (UniqueName: \"kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp\") pod \"maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:13.031076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.031010    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"]
Oct 31 00:31:13.033982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.033963    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9de170-pqhd4"]
Oct 31 00:31:13.095336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.095307    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:13.398051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.398011    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"]
Oct 31 00:31:13.401494 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:31:13.401447    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ead0a1_c5c9_4a85_af06_3f4a4498cd65.slice/crio-5d41bb1273c608fc44cee313141a1f1b36bcd746fe5ba88a90b0ee6deaa8c2f1 WatchSource:0}: Error finding container 5d41bb1273c608fc44cee313141a1f1b36bcd746fe5ba88a90b0ee6deaa8c2f1: Status 404 returned error can't find the container with id 5d41bb1273c608fc44cee313141a1f1b36bcd746fe5ba88a90b0ee6deaa8c2f1
Oct 31 00:31:13.818625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.818591    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:31:13.899432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.899396    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume\") pod \"72076e34-5810-4c87-ae9a-6bf9e3456b71\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") "
Oct 31 00:31:13.899644 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.899508    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg\" (UniqueName: \"kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg\") pod \"72076e34-5810-4c87-ae9a-6bf9e3456b71\" (UID: \"72076e34-5810-4c87-ae9a-6bf9e3456b71\") "
Oct 31 00:31:13.900130 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.900097    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume" pod "72076e34-5810-4c87-ae9a-6bf9e3456b71" (UID: "72076e34-5810-4c87-ae9a-6bf9e3456b71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:13.902148 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.902124    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg" pod "72076e34-5810-4c87-ae9a-6bf9e3456b71" (UID: "72076e34-5810-4c87-ae9a-6bf9e3456b71"). InnerVolumeSpecName "kube-api-access-254lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:13.986237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.986148    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5" event={"ID":"33ead0a1-c5c9-4a85-af06-3f4a4498cd65","Type":"ContainerStarted","Data":"06d0c0b2e448153d426e905937e67ea8b4774e3c0ec6999cb345dc7d35e28ddc"}
Oct 31 00:31:13.986237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.986215    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5" event={"ID":"33ead0a1-c5c9-4a85-af06-3f4a4498cd65","Type":"ContainerStarted","Data":"5d41bb1273c608fc44cee313141a1f1b36bcd746fe5ba88a90b0ee6deaa8c2f1"}
Oct 31 00:31:13.987824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.987801    5078 generic.go:356] "Generic (PLEG): container finished" podID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerID="503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5" exitCode=0
Oct 31 00:31:13.987954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.987842    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" event={"ID":"72076e34-5810-4c87-ae9a-6bf9e3456b71","Type":"ContainerDied","Data":"503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5"}
Oct 31 00:31:13.987954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.987904    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h" event={"ID":"72076e34-5810-4c87-ae9a-6bf9e3456b71","Type":"ContainerDied","Data":"4794cd1c6e448cae8d11bcf1727455dee43a3df80228d0d2722e086dd68d5abe"}
Oct 31 00:31:13.987954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.987853    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"
Oct 31 00:31:13.987954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:13.987940    5078 scope.go:122] "RemoveContainer" containerID="503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5"
Oct 31 00:31:14.001643 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.001615    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254lg\" (UniqueName: \"kubernetes.io/projected/72076e34-5810-4c87-ae9a-6bf9e3456b71-kube-api-access-254lg\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:14.001767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.001644    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076e34-5810-4c87-ae9a-6bf9e3456b71-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:14.009121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.009069    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5" podStartSLOduration=2.009052973 podStartE2EDuration="2.009052973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:14.006864089 +0000 UTC m=+2241.953234809" watchObservedRunningTime="2026-10-31 00:31:14.009052973 +0000 UTC m=+2241.955423691"
Oct 31 00:31:14.017795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.017774    5078 scope.go:122] "RemoveContainer" containerID="503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5"
Oct 31 00:31:14.018164 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:14.018139    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5\": container with ID starting with 503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5 not found: ID does not exist" containerID="503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5"
Oct 31 00:31:14.018218 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.018177    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5"} err="failed to get container status \"503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5\": rpc error: code = NotFound desc = could not find container \"503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5\": container with ID starting with 503d6831dedd4c443b85feed382185f22bed99d517aa7e3ffc08550b5e28dcc5 not found: ID does not exist"
Oct 31 00:31:14.031025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.030997    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"]
Oct 31 00:31:14.035072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.035051    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0402df-6469bd8c4c-z884h"]
Oct 31 00:31:14.197818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.197788    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" path="/var/lib/kubelet/pods/72076e34-5810-4c87-ae9a-6bf9e3456b71/volumes"
Oct 31 00:31:14.198236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.198221    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6" path="/var/lib/kubelet/pods/e986cf3c-6bc4-4a9e-9777-e51e1f45c2b6/volumes"
Oct 31 00:31:14.997397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.997365    5078 generic.go:356] "Generic (PLEG): container finished" podID="33ead0a1-c5c9-4a85-af06-3f4a4498cd65" containerID="06d0c0b2e448153d426e905937e67ea8b4774e3c0ec6999cb345dc7d35e28ddc" exitCode=0
Oct 31 00:31:14.997802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:14.997451    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5" event={"ID":"33ead0a1-c5c9-4a85-af06-3f4a4498cd65","Type":"ContainerDied","Data":"06d0c0b2e448153d426e905937e67ea8b4774e3c0ec6999cb345dc7d35e28ddc"}
Oct 31 00:31:15.000334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.000312    5078 generic.go:356] "Generic (PLEG): container finished" podID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerID="b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c" exitCode=0
Oct 31 00:31:15.000443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.000384    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" event={"ID":"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d","Type":"ContainerDied","Data":"b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"}
Oct 31 00:31:15.000815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.000801    5078 scope.go:122] "RemoveContainer" containerID="b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"
Oct 31 00:31:15.002178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.002154    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:31:15.002236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.002226    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:31:15.002273 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.002263    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:31:15.163609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.163574    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"]
Oct 31 00:31:15.163839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.163810    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" containerID="cri-o://c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c" gracePeriod=30
Oct 31 00:31:15.250047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.249944    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:31:15.250047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.249995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:31:15.465372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:15.465342    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"]
Oct 31 00:31:16.013306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.013268    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" event={"ID":"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d","Type":"ContainerStarted","Data":"34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077"}
Oct 31 00:31:16.013795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.013349    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy" containerID="cri-o://34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077" gracePeriod=30
Oct 31 00:31:16.186142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.186120    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:16.344420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.344334    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca\") pod \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") "
Oct 31 00:31:16.344612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.344505    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp\" (UniqueName: \"kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp\") pod \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") "
Oct 31 00:31:16.344612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.344537    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca\") pod \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\" (UID: \"33ead0a1-c5c9-4a85-af06-3f4a4498cd65\") "
Oct 31 00:31:16.345066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.344985    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca" pod "33ead0a1-c5c9-4a85-af06-3f4a4498cd65" (UID: "33ead0a1-c5c9-4a85-af06-3f4a4498cd65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:16.346829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.346798    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca" pod "33ead0a1-c5c9-4a85-af06-3f4a4498cd65" (UID: "33ead0a1-c5c9-4a85-af06-3f4a4498cd65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:16.347288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.347262    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp" pod "33ead0a1-c5c9-4a85-af06-3f4a4498cd65" (UID: "33ead0a1-c5c9-4a85-af06-3f4a4498cd65"). InnerVolumeSpecName "kube-api-access-sb2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:16.446320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.446285    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2xp\" (UniqueName: \"kubernetes.io/projected/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-kube-api-access-sb2xp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:16.446320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.446314    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:16.446320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.446325    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ead0a1-c5c9-4a85-af06-3f4a4498cd65-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:16.865633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:16.865591    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:31:17.021910 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:17.021883    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"
Oct 31 00:31:17.022321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:17.021907    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5" event={"ID":"33ead0a1-c5c9-4a85-af06-3f4a4498cd65","Type":"ContainerDied","Data":"5d41bb1273c608fc44cee313141a1f1b36bcd746fe5ba88a90b0ee6deaa8c2f1"}
Oct 31 00:31:17.022321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:17.021954    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d41bb1273c608fc44cee313141a1f1b36bcd746fe5ba88a90b0ee6deaa8c2f1"
Oct 31 00:31:17.459289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:17.459254    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"]
Oct 31 00:31:17.491041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:17.491007    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"]
Oct 31 00:31:20.471012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.470938    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:31:20.516357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516323    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516435    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516555    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516583    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516604    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516640    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516703    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.516985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516856    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:20.517102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.517075    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:20.517210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.517185    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:20.517389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.517214    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:20.517389 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.516734    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.517603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.517444    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket\") pod \"905c8138-3bf0-41f0-b1db-fa377aa449c3\" (UID: \"905c8138-3bf0-41f0-b1db-fa377aa449c3\") "
Oct 31 00:31:20.517808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.517784    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:20.518094 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.518070    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.518180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.518099    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.518180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.518113    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.518180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.518129    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.518180 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.518143    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905c8138-3bf0-41f0-b1db-fa377aa449c3-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.523232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.521634    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "kube-api-access-2xgmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:20.523232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.521672    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:20.523232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.521812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:20.524790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.524761    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo" pod "905c8138-3bf0-41f0-b1db-fa377aa449c3" (UID: "905c8138-3bf0-41f0-b1db-fa377aa449c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:31:20.619895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.619857    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xgmg\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-kube-api-access-2xgmg\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.619895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.619899    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.620173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.619914    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:20.620173 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:20.619927    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905c8138-3bf0-41f0-b1db-fa377aa449c3-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.052927 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.052896    5078 generic.go:356] "Generic (PLEG): container finished" podID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerID="c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c" exitCode=0
Oct 31 00:31:21.053083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.052948    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"
Oct 31 00:31:21.053083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.052985    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" event={"ID":"905c8138-3bf0-41f0-b1db-fa377aa449c3","Type":"ContainerDied","Data":"c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c"}
Oct 31 00:31:21.053083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.053068    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n" event={"ID":"905c8138-3bf0-41f0-b1db-fa377aa449c3","Type":"ContainerDied","Data":"e59aae555a9e2460d5024d4b51b89e7b885ef8aa7c627bcb7b33fd80cc7a8515"}
Oct 31 00:31:21.053237 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.053092    5078 scope.go:122] "RemoveContainer" containerID="c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c"
Oct 31 00:31:21.127754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.127720    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"]
Oct 31 00:31:21.130707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.130631    5078 scope.go:122] "RemoveContainer" containerID="c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c"
Oct 31 00:31:21.131026 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:21.130995    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c\": container with ID starting with c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c not found: ID does not exist" containerID="c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c"
Oct 31 00:31:21.131127 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.131026    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c"} err="failed to get container status \"c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c\": rpc error: code = NotFound desc = could not find container \"c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c\": container with ID starting with c37918f651b80d5362e7b8c9cba9a2ef05b70418ba04b64c35f4b335110ef33c not found: ID does not exist"
Oct 31 00:31:21.132980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.132957    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-jxb6n"]
Oct 31 00:31:21.297223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.297198    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:31:21.326841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.326810    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.326864    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.326914    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.326989    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.327034    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327189 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.327132    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.327190    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.327218    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.327296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.327274    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket\") pod \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\" (UID: \"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d\") "
Oct 31 00:31:21.328267 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.328236    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:21.329134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.328794    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:21.329134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.329093    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:21.329343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.329247    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:21.329525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.329454    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:21.332054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.332017    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:21.332915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.332890    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "kube-api-access-lpcwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:21.333041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.332967    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:21.334890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.334864    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo" pod "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" (UID: "3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:31:21.428972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.428933    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.428972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.428972    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.428986    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.428995    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.429005    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcwm\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-kube-api-access-lpcwm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.429015    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.429024    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.429032    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:21.429187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:21.429042    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:22.063409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.063371    5078 generic.go:356] "Generic (PLEG): container finished" podID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerID="34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077" exitCode=0
Oct 31 00:31:22.063936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.063430    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"
Oct 31 00:31:22.063936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.063495    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" event={"ID":"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d","Type":"ContainerDied","Data":"34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077"}
Oct 31 00:31:22.063936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.063569    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8" event={"ID":"3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d","Type":"ContainerDied","Data":"d90a17c634cdd09ec2f57b1a4cd79e6356e2f8070e2a7a31fb533403579c48f9"}
Oct 31 00:31:22.063936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.063606    5078 scope.go:122] "RemoveContainer" containerID="34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077"
Oct 31 00:31:22.075501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.075440    5078 scope.go:122] "RemoveContainer" containerID="b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"
Oct 31 00:31:22.085906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.085842    5078 scope.go:122] "RemoveContainer" containerID="34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077"
Oct 31 00:31:22.086170 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:22.086145    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077\": container with ID starting with 34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077 not found: ID does not exist" containerID="34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077"
Oct 31 00:31:22.086293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.086179    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077"} err="failed to get container status \"34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077\": rpc error: code = NotFound desc = could not find container \"34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077\": container with ID starting with 34c78568f3d10c15fe0fd0a82bbc12e2cabb9f030a517cbfe8f2cec73c475077 not found: ID does not exist"
Oct 31 00:31:22.086293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.086202    5078 scope.go:122] "RemoveContainer" containerID="b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"
Oct 31 00:31:22.086543 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:22.086515    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c\": container with ID starting with b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c not found: ID does not exist" containerID="b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"
Oct 31 00:31:22.086684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.086565    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c"} err="failed to get container status \"b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c\": rpc error: code = NotFound desc = could not find container \"b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c\": container with ID starting with b568563ab9ff73eb0f5292ffbdad36efb8958ee769ef06f803e66970459dde4c not found: ID does not exist"
Oct 31 00:31:22.090764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.090742    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"]
Oct 31 00:31:22.094795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.094775    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-945abd-openshift-default-db85747dc-57zh8"]
Oct 31 00:31:22.196502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.196474    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" path="/var/lib/kubelet/pods/3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d/volumes"
Oct 31 00:31:22.196972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.196959    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" path="/var/lib/kubelet/pods/905c8138-3bf0-41f0-b1db-fa377aa449c3/volumes"
Oct 31 00:31:22.939475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:22.939423    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"]
Oct 31 00:31:26.155383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.155350    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"]
Oct 31 00:31:26.167153 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.167125    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"]
Oct 31 00:31:26.487669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.487576    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"]
Oct 31 00:31:26.488417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488385    5078 cpu_manager.go:414] "containerMap: removing container" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing"
Oct 31 00:31:26.488417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488413    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488426    5078 cpu_manager.go:414] "containerMap: removing container" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488433    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488445    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488453    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488497    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488504    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488551    5078 cpu_manager.go:414] "containerMap: removing container" podUID="33ead0a1-c5c9-4a85-af06-3f4a4498cd65" containerName="revoke-keys"
Oct 31 00:31:26.488699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488558    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ead0a1-c5c9-4a85-af06-3f4a4498cd65" containerName="revoke-keys"
Oct 31 00:31:26.489155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488726    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="72076e34-5810-4c87-ae9a-6bf9e3456b71" containerName="payload-pre-processing"
Oct 31 00:31:26.489155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488738    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="905c8138-3bf0-41f0-b1db-fa377aa449c3" containerName="istio-proxy"
Oct 31 00:31:26.489155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488749    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy"
Oct 31 00:31:26.489155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.488764    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ead0a1-c5c9-4a85-af06-3f4a4498cd65" containerName="revoke-keys"
Oct 31 00:31:26.494191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.494165    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"]
Oct 31 00:31:26.494514 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.494491    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.507479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.507437    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:31:26.595939 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.595909    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:31:26.614700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.614667    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.614865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.614714    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.614865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.614841    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgf4\" (UniqueName: \"kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.715808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.715767    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgf4\" (UniqueName: \"kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.715981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.715965    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.716069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.716001    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.716710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.716684    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.718835 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.718807    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.730694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.730669    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgf4\" (UniqueName: \"kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4\") pod \"maas-api-revoke-sub-3bb1795f-6rv9n\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:26.829810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:26.829737    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:27.133334 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.133304    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"]
Oct 31 00:31:27.133503 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:31:27.133478    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6be281_7a75_4f13_948a_204d33efcb7f.slice/crio-781c980eb13b190832788792504117b31f9eacf3fc863fb96f2ed96ecca950e6 WatchSource:0}: Error finding container 781c980eb13b190832788792504117b31f9eacf3fc863fb96f2ed96ecca950e6: Status 404 returned error can't find the container with id 781c980eb13b190832788792504117b31f9eacf3fc863fb96f2ed96ecca950e6
Oct 31 00:31:27.567821 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.567792    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"]
Oct 31 00:31:27.568309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.568043    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" podUID="b32cf632-468b-4ffc-a143-4ff9d703a3e6" containerName="maas-api" containerID="cri-o://7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612" gracePeriod=30
Oct 31 00:31:27.874535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.866673    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:31:27.931853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.931822    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs\") pod \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") "
Oct 31 00:31:27.932055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.931890    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk\" (UniqueName: \"kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk\") pod \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") "
Oct 31 00:31:27.932129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.932081    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca\") pod \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") "
Oct 31 00:31:27.932183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.932138    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\") pod \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\" (UID: \"b32cf632-468b-4ffc-a143-4ff9d703a3e6\") "
Oct 31 00:31:27.932530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.932500    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca" pod "b32cf632-468b-4ffc-a143-4ff9d703a3e6" (UID: "b32cf632-468b-4ffc-a143-4ff9d703a3e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:27.932650 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.932626    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32cf632-468b-4ffc-a143-4ff9d703a3e6-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:27.934513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.934382    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs" pod "b32cf632-468b-4ffc-a143-4ff9d703a3e6" (UID: "b32cf632-468b-4ffc-a143-4ff9d703a3e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:27.934810 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.934789    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk" pod "b32cf632-468b-4ffc-a143-4ff9d703a3e6" (UID: "b32cf632-468b-4ffc-a143-4ff9d703a3e6"). InnerVolumeSpecName "kube-api-access-75dnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:27.936838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.936817    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls" pod "b32cf632-468b-4ffc-a143-4ff9d703a3e6" (UID: "b32cf632-468b-4ffc-a143-4ff9d703a3e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:27.940015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:27.939928    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" containerID="cri-o://b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837" gracePeriod=25
Oct 31 00:31:28.034086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.034052    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.034249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.034099    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32cf632-468b-4ffc-a143-4ff9d703a3e6-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.034249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.034117    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dnk\" (UniqueName: \"kubernetes.io/projected/b32cf632-468b-4ffc-a143-4ff9d703a3e6-kube-api-access-75dnk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.112909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.112871    5078 generic.go:356] "Generic (PLEG): container finished" podID="b32cf632-468b-4ffc-a143-4ff9d703a3e6" containerID="7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612" exitCode=0
Oct 31 00:31:28.113090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.112937    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"
Oct 31 00:31:28.113090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.112959    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" event={"ID":"b32cf632-468b-4ffc-a143-4ff9d703a3e6","Type":"ContainerDied","Data":"7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612"}
Oct 31 00:31:28.113090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.113010    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc" event={"ID":"b32cf632-468b-4ffc-a143-4ff9d703a3e6","Type":"ContainerDied","Data":"7fad2f70616e66183b768a42277d978d43a285714ac7f66dcae359b400fd2287"}
Oct 31 00:31:28.113090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.113041    5078 scope.go:122] "RemoveContainer" containerID="7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612"
Oct 31 00:31:28.115074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.115049    5078 generic.go:356] "Generic (PLEG): container finished" podID="ef6be281-7a75-4f13-948a-204d33efcb7f" containerID="f16c2d69d2ab8a3d2998dc773b904651e1b3ecb80558d0d7a49bcf74f1a107f7" exitCode=0
Oct 31 00:31:28.115209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.115127    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n" event={"ID":"ef6be281-7a75-4f13-948a-204d33efcb7f","Type":"ContainerDied","Data":"f16c2d69d2ab8a3d2998dc773b904651e1b3ecb80558d0d7a49bcf74f1a107f7"}
Oct 31 00:31:28.115209 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.115167    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n" event={"ID":"ef6be281-7a75-4f13-948a-204d33efcb7f","Type":"ContainerStarted","Data":"781c980eb13b190832788792504117b31f9eacf3fc863fb96f2ed96ecca950e6"}
Oct 31 00:31:28.125316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.125293    5078 scope.go:122] "RemoveContainer" containerID="7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612"
Oct 31 00:31:28.125613 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:28.125593    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612\": container with ID starting with 7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612 not found: ID does not exist" containerID="7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612"
Oct 31 00:31:28.125692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.125619    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612"} err="failed to get container status \"7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612\": rpc error: code = NotFound desc = could not find container \"7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612\": container with ID starting with 7ab304032b4c5b0a603d922d755ccacc0b07b072649e96c71c325cd557d31612 not found: ID does not exist"
Oct 31 00:31:28.158866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.158832    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"]
Oct 31 00:31:28.166588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.166568    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a8d8bf-6d49bfd5b5-hj7bc"]
Oct 31 00:31:28.211098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:28.211018    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32cf632-468b-4ffc-a143-4ff9d703a3e6" path="/var/lib/kubelet/pods/b32cf632-468b-4ffc-a143-4ff9d703a3e6/volumes"
Oct 31 00:31:29.288820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.288794    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:29.349977 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.349926    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4\" (UniqueName: \"kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4\") pod \"ef6be281-7a75-4f13-948a-204d33efcb7f\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") "
Oct 31 00:31:29.350219 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.350197    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca\") pod \"ef6be281-7a75-4f13-948a-204d33efcb7f\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") "
Oct 31 00:31:29.350704 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.350310    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca\") pod \"ef6be281-7a75-4f13-948a-204d33efcb7f\" (UID: \"ef6be281-7a75-4f13-948a-204d33efcb7f\") "
Oct 31 00:31:29.350865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.350788    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca" pod "ef6be281-7a75-4f13-948a-204d33efcb7f" (UID: "ef6be281-7a75-4f13-948a-204d33efcb7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:29.353597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.353552    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca" pod "ef6be281-7a75-4f13-948a-204d33efcb7f" (UID: "ef6be281-7a75-4f13-948a-204d33efcb7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:29.354155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.354131    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4" pod "ef6be281-7a75-4f13-948a-204d33efcb7f" (UID: "ef6be281-7a75-4f13-948a-204d33efcb7f"). InnerVolumeSpecName "kube-api-access-mbgf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:29.451759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.451733    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:29.451759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.451760    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be281-7a75-4f13-948a-204d33efcb7f-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:29.451956 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:29.451770    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbgf4\" (UniqueName: \"kubernetes.io/projected/ef6be281-7a75-4f13-948a-204d33efcb7f-kube-api-access-mbgf4\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:30.138143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:30.138115    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"
Oct 31 00:31:30.138341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:30.138149    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n" event={"ID":"ef6be281-7a75-4f13-948a-204d33efcb7f","Type":"ContainerDied","Data":"781c980eb13b190832788792504117b31f9eacf3fc863fb96f2ed96ecca950e6"}
Oct 31 00:31:30.138341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:30.138206    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781c980eb13b190832788792504117b31f9eacf3fc863fb96f2ed96ecca950e6"
Oct 31 00:31:31.156624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.156569    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs" podUID="7307d862-3d93-4cd1-99b4-2c46b74b3e44" containerName="payload-processing" containerID="cri-o://ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f" gracePeriod=25
Oct 31 00:31:31.168037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.168000    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj" podUID="f0db0989-f9e7-40d6-bf01-53587d33c3a4" containerName="payload-pre-processing" containerID="cri-o://3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd" gracePeriod=25
Oct 31 00:31:31.395284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.395235    5078 scope.go:122] "RemoveContainer" containerID="9c4985213ba78c3c7d29f81c56ef3c77326dae5b7c8ee00602e161481abbb144"
Oct 31 00:31:31.505535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.505512    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:31:31.536720 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.536696    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:31:31.581576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.581548    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume\") pod \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") "
Oct 31 00:31:31.581839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.581818    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2\" (UniqueName: \"kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2\") pod \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\" (UID: \"7307d862-3d93-4cd1-99b4-2c46b74b3e44\") "
Oct 31 00:31:31.582373 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.581962    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume" pod "7307d862-3d93-4cd1-99b4-2c46b74b3e44" (UID: "7307d862-3d93-4cd1-99b4-2c46b74b3e44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:31.583265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.583240    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7307d862-3d93-4cd1-99b4-2c46b74b3e44-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:31.584349 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.584322    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2" pod "7307d862-3d93-4cd1-99b4-2c46b74b3e44" (UID: "7307d862-3d93-4cd1-99b4-2c46b74b3e44"). InnerVolumeSpecName "kube-api-access-9k9r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:31.684276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.684192    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume\") pod \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") "
Oct 31 00:31:31.684276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.684242    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf\" (UniqueName: \"kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf\") pod \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\" (UID: \"f0db0989-f9e7-40d6-bf01-53587d33c3a4\") "
Oct 31 00:31:31.684666 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.684640    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume" pod "f0db0989-f9e7-40d6-bf01-53587d33c3a4" (UID: "f0db0989-f9e7-40d6-bf01-53587d33c3a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:31.684774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.684684    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9r2\" (UniqueName: \"kubernetes.io/projected/7307d862-3d93-4cd1-99b4-2c46b74b3e44-kube-api-access-9k9r2\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:31.686933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.686913    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf" pod "f0db0989-f9e7-40d6-bf01-53587d33c3a4" (UID: "f0db0989-f9e7-40d6-bf01-53587d33c3a4"). InnerVolumeSpecName "kube-api-access-cs5xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:31.785933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.785901    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db0989-f9e7-40d6-bf01-53587d33c3a4-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:31.785933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:31.785932    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5xf\" (UniqueName: \"kubernetes.io/projected/f0db0989-f9e7-40d6-bf01-53587d33c3a4-kube-api-access-cs5xf\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:32.153034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.152997    5078 generic.go:356] "Generic (PLEG): container finished" podID="7307d862-3d93-4cd1-99b4-2c46b74b3e44" containerID="ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f" exitCode=0
Oct 31 00:31:32.153234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.153044    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"
Oct 31 00:31:32.153234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.153080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs" event={"ID":"7307d862-3d93-4cd1-99b4-2c46b74b3e44","Type":"ContainerDied","Data":"ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f"}
Oct 31 00:31:32.153234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.153130    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs" event={"ID":"7307d862-3d93-4cd1-99b4-2c46b74b3e44","Type":"ContainerDied","Data":"a59f15a80a67ce8d319b33dfade331e6fecfec946ce55f2d4a7e4d9ebfe92137"}
Oct 31 00:31:32.153234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.153153    5078 scope.go:122] "RemoveContainer" containerID="ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f"
Oct 31 00:31:32.154751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.154707    5078 generic.go:356] "Generic (PLEG): container finished" podID="f0db0989-f9e7-40d6-bf01-53587d33c3a4" containerID="3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd" exitCode=0
Oct 31 00:31:32.154897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.154779    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"
Oct 31 00:31:32.154897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.154819    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj" event={"ID":"f0db0989-f9e7-40d6-bf01-53587d33c3a4","Type":"ContainerDied","Data":"3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd"}
Oct 31 00:31:32.155298 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.155266    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj" event={"ID":"f0db0989-f9e7-40d6-bf01-53587d33c3a4","Type":"ContainerDied","Data":"87dc67dfb400afc691e459d5917772e0aa649c7e6271dfa187a68dc52d2961d1"}
Oct 31 00:31:32.164057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.163893    5078 scope.go:122] "RemoveContainer" containerID="ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f"
Oct 31 00:31:32.164295 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:32.164204    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f\": container with ID starting with ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f not found: ID does not exist" containerID="ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f"
Oct 31 00:31:32.164295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.164240    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f"} err="failed to get container status \"ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f\": rpc error: code = NotFound desc = could not find container \"ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f\": container with ID starting with ed044235e9faff1d76d4cb19bd807abbf341aaa1ecab7c742356f38d4570cc9f not found: ID does not exist"
Oct 31 00:31:32.164295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.164264    5078 scope.go:122] "RemoveContainer" containerID="3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd"
Oct 31 00:31:32.182302 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.182277    5078 scope.go:122] "RemoveContainer" containerID="3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd"
Oct 31 00:31:32.182653 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:32.182626    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd\": container with ID starting with 3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd not found: ID does not exist" containerID="3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd"
Oct 31 00:31:32.182734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.182665    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd"} err="failed to get container status \"3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd\": rpc error: code = NotFound desc = could not find container \"3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd\": container with ID starting with 3907e64156516ae92b8692bf0f5fffb52d348006a1da40d3ea125339ad4d7cfd not found: ID does not exist"
Oct 31 00:31:32.197083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.196265    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"]
Oct 31 00:31:32.202431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.202404    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a8d8bf-7ccf759fc9-cl6fs"]
Oct 31 00:31:32.220505 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.220474    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"]
Oct 31 00:31:32.225615 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.225589    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a8d8bf-7dccb4fc6c-4sdfj"]
Oct 31 00:31:32.481288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.481192    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main" containerID="cri-o://56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406" gracePeriod=45
Oct 31 00:31:32.512903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.512863    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main" containerID="cri-o://76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb" gracePeriod=45
Oct 31 00:31:32.881767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.881736    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:31:32.938689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:32.938657    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:31:33.001917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.001821    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache\") pod \"641786fc-cd71-417f-a8b2-153079f8b512\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") "
Oct 31 00:31:33.002092 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.001982    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs\" (UniqueName: \"kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs\") pod \"641786fc-cd71-417f-a8b2-153079f8b512\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") "
Oct 31 00:31:33.002164 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.002105    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home\") pod \"641786fc-cd71-417f-a8b2-153079f8b512\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") "
Oct 31 00:31:33.002239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.002211    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk\" (UniqueName: \"kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk\") pod \"641786fc-cd71-417f-a8b2-153079f8b512\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") "
Oct 31 00:31:33.002384 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.002299    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir\") pod \"641786fc-cd71-417f-a8b2-153079f8b512\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") "
Oct 31 00:31:33.002384 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.002352    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm\") pod \"641786fc-cd71-417f-a8b2-153079f8b512\" (UID: \"641786fc-cd71-417f-a8b2-153079f8b512\") "
Oct 31 00:31:33.004546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.004324    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home" pod "641786fc-cd71-417f-a8b2-153079f8b512" (UID: "641786fc-cd71-417f-a8b2-153079f8b512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.004663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.004560    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache" pod "641786fc-cd71-417f-a8b2-153079f8b512" (UID: "641786fc-cd71-417f-a8b2-153079f8b512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.005500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.005417    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir" pod "641786fc-cd71-417f-a8b2-153079f8b512" (UID: "641786fc-cd71-417f-a8b2-153079f8b512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.006816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.006770    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm" pod "641786fc-cd71-417f-a8b2-153079f8b512" (UID: "641786fc-cd71-417f-a8b2-153079f8b512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.007177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.007149    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs" pod "641786fc-cd71-417f-a8b2-153079f8b512" (UID: "641786fc-cd71-417f-a8b2-153079f8b512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:33.007603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.007583    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk" pod "641786fc-cd71-417f-a8b2-153079f8b512" (UID: "641786fc-cd71-417f-a8b2-153079f8b512"). InnerVolumeSpecName "kube-api-access-l27mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.112882    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm\") pod \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") "
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.112996    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs\") pod \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") "
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113038    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir\") pod \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") "
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113098    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn\" (UniqueName: \"kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn\") pod \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") "
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113159    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache\") pod \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") "
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113200    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home\") pod \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\" (UID: \"6f13311b-d0f7-4d27-96ee-7e6eca986f43\") "
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113672    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27mk\" (UniqueName: \"kubernetes.io/projected/641786fc-cd71-417f-a8b2-153079f8b512-kube-api-access-l27mk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113694    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113708    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113721    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113735    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641786fc-cd71-417f-a8b2-153079f8b512-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.113748    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641786fc-cd71-417f-a8b2-153079f8b512-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.114378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.114042    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home" pod "6f13311b-d0f7-4d27-96ee-7e6eca986f43" (UID: "6f13311b-d0f7-4d27-96ee-7e6eca986f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.121220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.120894    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm" pod "6f13311b-d0f7-4d27-96ee-7e6eca986f43" (UID: "6f13311b-d0f7-4d27-96ee-7e6eca986f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.124766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.124452    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir" pod "6f13311b-d0f7-4d27-96ee-7e6eca986f43" (UID: "6f13311b-d0f7-4d27-96ee-7e6eca986f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.125084 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.124975    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache" pod "6f13311b-d0f7-4d27-96ee-7e6eca986f43" (UID: "6f13311b-d0f7-4d27-96ee-7e6eca986f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:33.125638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.125601    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs" pod "6f13311b-d0f7-4d27-96ee-7e6eca986f43" (UID: "6f13311b-d0f7-4d27-96ee-7e6eca986f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:33.128726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.128688    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn" pod "6f13311b-d0f7-4d27-96ee-7e6eca986f43" (UID: "6f13311b-d0f7-4d27-96ee-7e6eca986f43"). InnerVolumeSpecName "kube-api-access-cvnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:33.163998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.163957    5078 generic.go:356] "Generic (PLEG): container finished" podID="641786fc-cd71-417f-a8b2-153079f8b512" containerID="76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb" exitCode=0
Oct 31 00:31:33.164190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.164057    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"
Oct 31 00:31:33.164190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.164078    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" event={"ID":"641786fc-cd71-417f-a8b2-153079f8b512","Type":"ContainerDied","Data":"76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb"}
Oct 31 00:31:33.164190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.164133    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk" event={"ID":"641786fc-cd71-417f-a8b2-153079f8b512","Type":"ContainerDied","Data":"acc50cb40b40f2629acc3da6542ae637cc3cded8bbbce21b7aadeeaa05875390"}
Oct 31 00:31:33.164190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.164160    5078 scope.go:122] "RemoveContainer" containerID="76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb"
Oct 31 00:31:33.168179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.168155    5078 generic.go:356] "Generic (PLEG): container finished" podID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerID="56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406" exitCode=0
Oct 31 00:31:33.168310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.168196    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"
Oct 31 00:31:33.168310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.168254    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" event={"ID":"6f13311b-d0f7-4d27-96ee-7e6eca986f43","Type":"ContainerDied","Data":"56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406"}
Oct 31 00:31:33.168310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.168294    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww" event={"ID":"6f13311b-d0f7-4d27-96ee-7e6eca986f43","Type":"ContainerDied","Data":"f68f264c4304063c856cef065f35200784e29c41e74e94a1e18b29a714098bfb"}
Oct 31 00:31:33.199052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.199030    5078 scope.go:122] "RemoveContainer" containerID="76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb"
Oct 31 00:31:33.199446 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:33.199391    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb\": container with ID starting with 76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb not found: ID does not exist" containerID="76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb"
Oct 31 00:31:33.199446 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.199424    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb"} err="failed to get container status \"76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb\": rpc error: code = NotFound desc = could not find container \"76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb\": container with ID starting with 76dc5edfeef16876beefaa577a3fc143a6214a8951d0bb62a0996cdf11e372bb not found: ID does not exist"
Oct 31 00:31:33.199648 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.199451    5078 scope.go:122] "RemoveContainer" containerID="56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406"
Oct 31 00:31:33.209549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.209528    5078 scope.go:122] "RemoveContainer" containerID="56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406"
Oct 31 00:31:33.209840 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:33.209823    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406\": container with ID starting with 56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406 not found: ID does not exist" containerID="56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406"
Oct 31 00:31:33.209915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.209848    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406"} err="failed to get container status \"56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406\": rpc error: code = NotFound desc = could not find container \"56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406\": container with ID starting with 56450811c380373759a1d38935ffafea2b395cf9a2d35828b48bdcc4ed74f406 not found: ID does not exist"
Oct 31 00:31:33.214578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.214555    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.214578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.214577    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.214747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.214586    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.214747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.214594    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnsn\" (UniqueName: \"kubernetes.io/projected/6f13311b-d0f7-4d27-96ee-7e6eca986f43-kube-api-access-cvnsn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.214747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.214604    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.214747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.214613    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f13311b-d0f7-4d27-96ee-7e6eca986f43-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:33.244118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.244089    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"]
Oct 31 00:31:33.300662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.300565    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-w2-7c5da3-kserve-6c4b446d88-pzqww"]
Oct 31 00:31:33.339511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.339453    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"]
Oct 31 00:31:33.364683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:33.364651    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/simulator-w2-7c5da3-kserve-c6f999bd4-lw5wk"]
Oct 31 00:31:34.208844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:34.208799    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641786fc-cd71-417f-a8b2-153079f8b512" path="/var/lib/kubelet/pods/641786fc-cd71-417f-a8b2-153079f8b512/volumes"
Oct 31 00:31:34.209365 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:34.209352    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" path="/var/lib/kubelet/pods/6f13311b-d0f7-4d27-96ee-7e6eca986f43/volumes"
Oct 31 00:31:34.210571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:34.209994    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7307d862-3d93-4cd1-99b4-2c46b74b3e44" path="/var/lib/kubelet/pods/7307d862-3d93-4cd1-99b4-2c46b74b3e44/volumes"
Oct 31 00:31:34.211583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:34.211563    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0db0989-f9e7-40d6-bf01-53587d33c3a4" path="/var/lib/kubelet/pods/f0db0989-f9e7-40d6-bf01-53587d33c3a4/volumes"
Oct 31 00:31:36.404758 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:36.404714    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:31:37.692933 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:37.692899    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:38.317568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.317529    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"]
Oct 31 00:31:38.318266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318244    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main"
Oct 31 00:31:38.318331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318269    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main"
Oct 31 00:31:38.318331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318288    5078 cpu_manager.go:414] "containerMap: removing container" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main"
Oct 31 00:31:38.318331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318295    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main"
Oct 31 00:31:38.318331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318308    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ef6be281-7a75-4f13-948a-204d33efcb7f" containerName="revoke-keys"
Oct 31 00:31:38.318331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318316    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6be281-7a75-4f13-948a-204d33efcb7f" containerName="revoke-keys"
Oct 31 00:31:38.318529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318352    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f0db0989-f9e7-40d6-bf01-53587d33c3a4" containerName="payload-pre-processing"
Oct 31 00:31:38.318529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318360    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0db0989-f9e7-40d6-bf01-53587d33c3a4" containerName="payload-pre-processing"
Oct 31 00:31:38.318529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318392    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b32cf632-468b-4ffc-a143-4ff9d703a3e6" containerName="maas-api"
Oct 31 00:31:38.318529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318398    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32cf632-468b-4ffc-a143-4ff9d703a3e6" containerName="maas-api"
Oct 31 00:31:38.318529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318417    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7307d862-3d93-4cd1-99b4-2c46b74b3e44" containerName="payload-processing"
Oct 31 00:31:38.318529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318424    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7307d862-3d93-4cd1-99b4-2c46b74b3e44" containerName="payload-processing"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318559    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32cf632-468b-4ffc-a143-4ff9d703a3e6" containerName="maas-api"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318571    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7307d862-3d93-4cd1-99b4-2c46b74b3e44" containerName="payload-processing"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318583    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f13311b-d0f7-4d27-96ee-7e6eca986f43" containerName="main"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318593    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="641786fc-cd71-417f-a8b2-153079f8b512" containerName="main"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318604    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6be281-7a75-4f13-948a-204d33efcb7f" containerName="revoke-keys"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318616    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edf2d39-3eb8-42e3-9e4a-2ab830a71f9d" containerName="istio-proxy"
Oct 31 00:31:38.318722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.318624    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0db0989-f9e7-40d6-bf01-53587d33c3a4" containerName="payload-pre-processing"
Oct 31 00:31:38.324081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.324061    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.331421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.331394    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"]
Oct 31 00:31:38.396597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.396559    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:31:38.396771 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.396608    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:31:38.406905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.406884    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:31:38.406905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.406885    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:31:38.481329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.481292    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.481520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.481358    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gscbp\" (UniqueName: \"kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.481602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.481579    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.583348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.583244    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.583348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.583325    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gscbp\" (UniqueName: \"kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.583639 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.583495    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.584110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.584074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.586732 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.586702    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.599051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.599015    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gscbp\" (UniqueName: \"kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp\") pod \"maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.641980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.641954    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:38.952858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:38.952830    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"]
Oct 31 00:31:38.954343 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:31:38.954315    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6afdcd4_916a_43ba_a6d7_3ce4b1879ec9.slice/crio-098f48c84dc3267faae8c799fd1f1bc3d81d57f9c3c16033b78e712090166464 WatchSource:0}: Error finding container 098f48c84dc3267faae8c799fd1f1bc3d81d57f9c3c16033b78e712090166464: Status 404 returned error can't find the container with id 098f48c84dc3267faae8c799fd1f1bc3d81d57f9c3c16033b78e712090166464
Oct 31 00:31:39.216985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:39.216892    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s" event={"ID":"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9","Type":"ContainerStarted","Data":"098f48c84dc3267faae8c799fd1f1bc3d81d57f9c3c16033b78e712090166464"}
Oct 31 00:31:39.526037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:39.525936    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"]
Oct 31 00:31:39.528566 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:39.528538    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7c5da3-wtrt5"]
Oct 31 00:31:40.195508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:40.195401    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ead0a1-c5c9-4a85-af06-3f4a4498cd65" path="/var/lib/kubelet/pods/33ead0a1-c5c9-4a85-af06-3f4a4498cd65/volumes"
Oct 31 00:31:40.224732 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:40.224699    5078 generic.go:356] "Generic (PLEG): container finished" podID="a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" containerID="931652e4201625e5f63db19f4e12cf1f8abeab02fd97c7af62144cb03826a2e4" exitCode=0
Oct 31 00:31:40.224893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:40.224773    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s" event={"ID":"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9","Type":"ContainerDied","Data":"931652e4201625e5f63db19f4e12cf1f8abeab02fd97c7af62144cb03826a2e4"}
Oct 31 00:31:41.406828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.406797    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:41.529893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.529805    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca\") pod \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") "
Oct 31 00:31:41.529893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.529882    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp\" (UniqueName: \"kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp\") pod \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") "
Oct 31 00:31:41.530147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.530036    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca\") pod \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\" (UID: \"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9\") "
Oct 31 00:31:41.530351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.530323    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca" pod "a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" (UID: "a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:41.530523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.530499    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:41.532142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.532114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca" pod "a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" (UID: "a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:41.532232 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.532213    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp" pod "a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" (UID: "a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9"). InnerVolumeSpecName "kube-api-access-gscbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:41.631357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.631319    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:41.631357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:41.631363    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gscbp\" (UniqueName: \"kubernetes.io/projected/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9-kube-api-access-gscbp\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:42.242689 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:42.242661    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"
Oct 31 00:31:42.242857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:42.242691    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s" event={"ID":"a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9","Type":"ContainerDied","Data":"098f48c84dc3267faae8c799fd1f1bc3d81d57f9c3c16033b78e712090166464"}
Oct 31 00:31:42.242857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:42.242745    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="098f48c84dc3267faae8c799fd1f1bc3d81d57f9c3c16033b78e712090166464"
Oct 31 00:31:43.106118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:43.106081    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"]
Oct 31 00:31:43.125138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:43.125107    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"]
Oct 31 00:31:47.693353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:47.693305    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:48.100628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.100513    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"]
Oct 31 00:31:48.101662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.101629    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy" containerID="cri-o://acb15657ba21ee15916d3dc028686548320ba5547ebd5fb4f1322bbbe93a264e" gracePeriod=30
Oct 31 00:31:48.104574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.104551    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"]
Oct 31 00:31:48.104841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.104796    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy" containerID="cri-o://e64dda186588cf6928fa3f315c023d5e6bcd5c045bb5e83c21f38cda3401cfa8" gracePeriod=30
Oct 31 00:31:48.274983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.274955    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:31:48.285015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.284987    5078 generic.go:356] "Generic (PLEG): container finished" podID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerID="b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837" exitCode=0
Oct 31 00:31:48.285147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.285049    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"
Oct 31 00:31:48.285147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.285124    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" event={"ID":"1ff2da4c-667d-4f1d-8994-0a2a972b43b3","Type":"ContainerDied","Data":"b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837"}
Oct 31 00:31:48.285250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.285185    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk" event={"ID":"1ff2da4c-667d-4f1d-8994-0a2a972b43b3","Type":"ContainerDied","Data":"18bce8d07bfcd89ff2abae41bd3b2ae3ffbaa1cb5f3fcd7397a3f920c3e59dfe"}
Oct 31 00:31:48.285250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.285217    5078 scope.go:122] "RemoveContainer" containerID="b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837"
Oct 31 00:31:48.298065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.298048    5078 scope.go:122] "RemoveContainer" containerID="b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837"
Oct 31 00:31:48.298385 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:31:48.298356    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837\": container with ID starting with b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837 not found: ID does not exist" containerID="b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837"
Oct 31 00:31:48.298476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.298387    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837"} err="failed to get container status \"b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837\": rpc error: code = NotFound desc = could not find container \"b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837\": container with ID starting with b4dbf07f2cf3d8dee850f720326190ff78467e45524506028ef1061dec7a0837 not found: ID does not exist"
Oct 31 00:31:48.304667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.304646    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume\") pod \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") "
Oct 31 00:31:48.304767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.304691    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258\" (UniqueName: \"kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258\") pod \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\" (UID: \"1ff2da4c-667d-4f1d-8994-0a2a972b43b3\") "
Oct 31 00:31:48.304972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.304951    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume" pod "1ff2da4c-667d-4f1d-8994-0a2a972b43b3" (UID: "1ff2da4c-667d-4f1d-8994-0a2a972b43b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:48.305102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.305083    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:48.307192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.307160    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258" pod "1ff2da4c-667d-4f1d-8994-0a2a972b43b3" (UID: "1ff2da4c-667d-4f1d-8994-0a2a972b43b3"). InnerVolumeSpecName "kube-api-access-ff258". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:48.406901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.406867    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff258\" (UniqueName: \"kubernetes.io/projected/1ff2da4c-667d-4f1d-8994-0a2a972b43b3-kube-api-access-ff258\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:48.616337 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.616309    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"]
Oct 31 00:31:48.632555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.632528    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7c5da3-5d47cc5679-qf2sk"]
Oct 31 00:31:48.923390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.923352    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"]
Oct 31 00:31:48.923902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.923887    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" containerName="revoke-keys"
Oct 31 00:31:48.923954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.923904    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" containerName="revoke-keys"
Oct 31 00:31:48.923954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.923925    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing"
Oct 31 00:31:48.923954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.923930    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing"
Oct 31 00:31:48.924057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.924046    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" containerName="revoke-keys"
Oct 31 00:31:48.924057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.924055    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" containerName="payload-pre-processing"
Oct 31 00:31:48.928840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.928820    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:48.940342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.940317    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"]
Oct 31 00:31:48.948658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.948621    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:31:48.948831 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.948721    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:31:48.959111 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.959080    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:31:48.959881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:48.959672    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:31:49.013228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.013188    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.013405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.013247    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkxg\" (UniqueName: \"kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.013405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.013349    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.114585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.114548    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkxg\" (UniqueName: \"kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.114799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.114597    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.114799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.114739    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.115281 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.115256    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.117395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.117366    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.129492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.129435    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkxg\" (UniqueName: \"kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg\") pod \"maas-api-revoke-sub-10a0e3f3-fsg8w\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.269902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.269826    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:49.553063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:49.553033    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"]
Oct 31 00:31:49.553606 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:31:49.553580    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3026d1d2_6db0_4b98_a40d_f0486afa9ef9.slice/crio-06aabf014fa5c0b8cf16b894b0a920213ea247d88960288c80bd48c2a41dc372 WatchSource:0}: Error finding container 06aabf014fa5c0b8cf16b894b0a920213ea247d88960288c80bd48c2a41dc372: Status 404 returned error can't find the container with id 06aabf014fa5c0b8cf16b894b0a920213ea247d88960288c80bd48c2a41dc372
Oct 31 00:31:50.200738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:50.200702    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff2da4c-667d-4f1d-8994-0a2a972b43b3" path="/var/lib/kubelet/pods/1ff2da4c-667d-4f1d-8994-0a2a972b43b3/volumes"
Oct 31 00:31:50.303519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:50.303483    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w" event={"ID":"3026d1d2-6db0-4b98-a40d-f0486afa9ef9","Type":"ContainerStarted","Data":"ea7eb9d89216e3b80345c378d159f0f4b5595bb3e31b0f544c7d0d784d6c7df6"}
Oct 31 00:31:50.303687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:50.303535    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w" event={"ID":"3026d1d2-6db0-4b98-a40d-f0486afa9ef9","Type":"ContainerStarted","Data":"06aabf014fa5c0b8cf16b894b0a920213ea247d88960288c80bd48c2a41dc372"}
Oct 31 00:31:50.333297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:50.333236    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w" podStartSLOduration=2.333221406 podStartE2EDuration="2.333221406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:50.331357434 +0000 UTC m=+2278.277728144" watchObservedRunningTime="2026-10-31 00:31:50.333221406 +0000 UTC m=+2278.279592124"
Oct 31 00:31:51.315511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:51.315437    5078 generic.go:356] "Generic (PLEG): container finished" podID="3026d1d2-6db0-4b98-a40d-f0486afa9ef9" containerID="ea7eb9d89216e3b80345c378d159f0f4b5595bb3e31b0f544c7d0d784d6c7df6" exitCode=0
Oct 31 00:31:51.316000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:51.315508    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w" event={"ID":"3026d1d2-6db0-4b98-a40d-f0486afa9ef9","Type":"ContainerDied","Data":"ea7eb9d89216e3b80345c378d159f0f4b5595bb3e31b0f544c7d0d784d6c7df6"}
Oct 31 00:31:52.485384 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.485357    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:52.553147 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.553110    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca\") pod \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") "
Oct 31 00:31:52.553332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.553166    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca\") pod \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") "
Oct 31 00:31:52.553332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.553214    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg\" (UniqueName: \"kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg\") pod \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\" (UID: \"3026d1d2-6db0-4b98-a40d-f0486afa9ef9\") "
Oct 31 00:31:52.553631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.553607    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca" pod "3026d1d2-6db0-4b98-a40d-f0486afa9ef9" (UID: "3026d1d2-6db0-4b98-a40d-f0486afa9ef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:52.555822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.555794    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg" pod "3026d1d2-6db0-4b98-a40d-f0486afa9ef9" (UID: "3026d1d2-6db0-4b98-a40d-f0486afa9ef9"). InnerVolumeSpecName "kube-api-access-xbkxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:52.558157 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.558133    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:52.560668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.560642    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca" pod "3026d1d2-6db0-4b98-a40d-f0486afa9ef9" (UID: "3026d1d2-6db0-4b98-a40d-f0486afa9ef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:52.659663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.659604    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:52.659663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:52.659661    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkxg\" (UniqueName: \"kubernetes.io/projected/3026d1d2-6db0-4b98-a40d-f0486afa9ef9-kube-api-access-xbkxg\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.332289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.332261    5078 generic.go:356] "Generic (PLEG): container finished" podID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerID="e64dda186588cf6928fa3f315c023d5e6bcd5c045bb5e83c21f38cda3401cfa8" exitCode=0
Oct 31 00:31:53.332455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.332335    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" event={"ID":"cd56ff0f-da7a-4de9-b2a0-125e471de6d2","Type":"ContainerDied","Data":"e64dda186588cf6928fa3f315c023d5e6bcd5c045bb5e83c21f38cda3401cfa8"}
Oct 31 00:31:53.332455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.332408    5078 scope.go:122] "RemoveContainer" containerID="edad74b1e307270b0d17d11a44f9221935caeb1c1df5382db1f63c3fd0f37305"
Oct 31 00:31:53.335003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.334976    5078 generic.go:356] "Generic (PLEG): container finished" podID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerID="acb15657ba21ee15916d3dc028686548320ba5547ebd5fb4f1322bbbe93a264e" exitCode=0
Oct 31 00:31:53.335110 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.335049    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" event={"ID":"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4","Type":"ContainerDied","Data":"acb15657ba21ee15916d3dc028686548320ba5547ebd5fb4f1322bbbe93a264e"}
Oct 31 00:31:53.336823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.336804    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w" event={"ID":"3026d1d2-6db0-4b98-a40d-f0486afa9ef9","Type":"ContainerDied","Data":"06aabf014fa5c0b8cf16b894b0a920213ea247d88960288c80bd48c2a41dc372"}
Oct 31 00:31:53.336950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.336835    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"
Oct 31 00:31:53.336950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.336836    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06aabf014fa5c0b8cf16b894b0a920213ea247d88960288c80bd48c2a41dc372"
Oct 31 00:31:53.385797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.385770    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:31:53.431211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.430803    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:31:53.469622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469594    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.469747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469638    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.469747 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469686    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.469840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469777    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.469840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469824    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.469921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469861    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.469921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469903    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469941    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.469975    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470039 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470031    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.470213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470072    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470111    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.470213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470140    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470183    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.470412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token\") pod \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\" (UID: \"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4\") "
Oct 31 00:31:53.470412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470286    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470320    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.470412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.470371    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket\") pod \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\" (UID: \"cd56ff0f-da7a-4de9-b2a0-125e471de6d2\") "
Oct 31 00:31:53.471098 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.471071    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.472735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.471501    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.472735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.471646    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.472735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.471786    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.472735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.472172    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.472735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.472246    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:53.472735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.472489    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.473451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.473176    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.473864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.473812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.474033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.474003    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:53.485843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.485803    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.485843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.485803    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "kube-api-access-swscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:53.489260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.489163    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:31:53.489260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.489223    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:53.489610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.489524    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6" pod "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" (UID: "9d43026d-495b-4d5c-b30c-4ee2f68bb8f4"). InnerVolumeSpecName "kube-api-access-gx5w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:53.489610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.489534    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:53.489610 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.489591    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:53.492181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.492057    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo" pod "cd56ff0f-da7a-4de9-b2a0-125e471de6d2" (UID: "cd56ff0f-da7a-4de9-b2a0-125e471de6d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:31:53.571873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571837    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.571873 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571872    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571886    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571900    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571914    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571927    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571941    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swscj\" (UniqueName: \"kubernetes.io/projected/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-kube-api-access-swscj\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571955    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571969    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571983    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.571996    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572010    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572023    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572035    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd56ff0f-da7a-4de9-b2a0-125e471de6d2-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572048    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572061    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572075    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5w6\" (UniqueName: \"kubernetes.io/projected/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-kube-api-access-gx5w6\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:53.572582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:53.572088    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:54.345248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.345215    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"
Oct 31 00:31:54.345248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.345230    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j" event={"ID":"cd56ff0f-da7a-4de9-b2a0-125e471de6d2","Type":"ContainerDied","Data":"9256a64ae94df3e36f50386e5596fec183a594cba6b1b7b8a9e95cf9dfb8693c"}
Oct 31 00:31:54.345533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.345298    5078 scope.go:122] "RemoveContainer" containerID="e64dda186588cf6928fa3f315c023d5e6bcd5c045bb5e83c21f38cda3401cfa8"
Oct 31 00:31:54.346853 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.346828    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl" event={"ID":"9d43026d-495b-4d5c-b30c-4ee2f68bb8f4","Type":"ContainerDied","Data":"7f69cb793cbdfb7bdd80840dbca90f5665f20550233a54aeb242facc3323e7be"}
Oct 31 00:31:54.346982 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.346833    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"
Oct 31 00:31:54.358663 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.358639    5078 scope.go:122] "RemoveContainer" containerID="acb15657ba21ee15916d3dc028686548320ba5547ebd5fb4f1322bbbe93a264e"
Oct 31 00:31:54.437107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.437073    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"]
Oct 31 00:31:54.445967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.445939    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-9w77j"]
Oct 31 00:31:54.479617 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.479513    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"]
Oct 31 00:31:54.485413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:54.485384    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-8g9tl"]
Oct 31 00:31:56.195091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:56.195051    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" path="/var/lib/kubelet/pods/9d43026d-495b-4d5c-b30c-4ee2f68bb8f4/volumes"
Oct 31 00:31:56.195682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:56.195664    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" path="/var/lib/kubelet/pods/cd56ff0f-da7a-4de9-b2a0-125e471de6d2/volumes"
Oct 31 00:31:58.129993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.129953    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main" containerID="cri-o://19f4559f3fe68403e292c5c93282060e525effc83e79d616d4672e16929314d7" gracePeriod=45
Oct 31 00:31:58.148199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.148168    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main" containerID="cri-o://2ceb9a86170ccb669d35be64fbdd04adf0494932305afc3a62fda264aea95901" gracePeriod=45
Oct 31 00:31:58.382591 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.382512    5078 generic.go:356] "Generic (PLEG): container finished" podID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerID="19f4559f3fe68403e292c5c93282060e525effc83e79d616d4672e16929314d7" exitCode=0
Oct 31 00:31:58.382740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.382650    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" event={"ID":"70a9ea32-a1a7-41eb-b335-1203dc85faaa","Type":"ContainerDied","Data":"19f4559f3fe68403e292c5c93282060e525effc83e79d616d4672e16929314d7"}
Oct 31 00:31:58.395844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.392456    5078 generic.go:356] "Generic (PLEG): container finished" podID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerID="2ceb9a86170ccb669d35be64fbdd04adf0494932305afc3a62fda264aea95901" exitCode=0
Oct 31 00:31:58.395844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.392581    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" event={"ID":"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f","Type":"ContainerDied","Data":"2ceb9a86170ccb669d35be64fbdd04adf0494932305afc3a62fda264aea95901"}
Oct 31 00:31:58.435424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.435400    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:31:58.442796 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.442772    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:31:58.525722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525678    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home\") pod \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") "
Oct 31 00:31:58.525909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525742    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs\") pod \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") "
Oct 31 00:31:58.525909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525780    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm\" (UniqueName: \"kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm\") pod \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") "
Oct 31 00:31:58.525909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525823    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir\") pod \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") "
Oct 31 00:31:58.525909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525843    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache\") pod \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") "
Oct 31 00:31:58.525909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525871    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs\") pod \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") "
Oct 31 00:31:58.525909 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525905    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir\") pod \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") "
Oct 31 00:31:58.526310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525937    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm\") pod \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") "
Oct 31 00:31:58.526310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.525970    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache\") pod \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") "
Oct 31 00:31:58.526310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526060    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home" pod "70a9ea32-a1a7-41eb-b335-1203dc85faaa" (UID: "70a9ea32-a1a7-41eb-b335-1203dc85faaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.526310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526093    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb\" (UniqueName: \"kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb\") pod \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") "
Oct 31 00:31:58.526310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526171    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm\") pod \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\" (UID: \"70a9ea32-a1a7-41eb-b335-1203dc85faaa\") "
Oct 31 00:31:58.526310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526207    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home\") pod \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\" (UID: \"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f\") "
Oct 31 00:31:58.526676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526484    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir" pod "70a9ea32-a1a7-41eb-b335-1203dc85faaa" (UID: "70a9ea32-a1a7-41eb-b335-1203dc85faaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.526676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526563    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache" pod "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" (UID: "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.526676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526575    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir" pod "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" (UID: "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.526676 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526584    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache" pod "70a9ea32-a1a7-41eb-b335-1203dc85faaa" (UID: "70a9ea32-a1a7-41eb-b335-1203dc85faaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.526901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526696    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home" pod "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" (UID: "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.526901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526713    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.526901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526732    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.526901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526750    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-model-cache\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.526901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526763    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.526901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.526776    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tmp-dir\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.529089 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.529055    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs" pod "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" (UID: "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:58.529501 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.529455    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm" pod "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" (UID: "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.529626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.529524    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb" pod "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" (UID: "0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f"). InnerVolumeSpecName "kube-api-access-6gmjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:58.530520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.530450    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm" pod "70a9ea32-a1a7-41eb-b335-1203dc85faaa" (UID: "70a9ea32-a1a7-41eb-b335-1203dc85faaa"). InnerVolumeSpecName "kube-api-access-bb9pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:58.530902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.530871    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm" pod "70a9ea32-a1a7-41eb-b335-1203dc85faaa" (UID: "70a9ea32-a1a7-41eb-b335-1203dc85faaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:58.531841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.531812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs" pod "70a9ea32-a1a7-41eb-b335-1203dc85faaa" (UID: "70a9ea32-a1a7-41eb-b335-1203dc85faaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:58.627970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.627939    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmjb\" (UniqueName: \"kubernetes.io/projected/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-kube-api-access-6gmjb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.627970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.627967    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a9ea32-a1a7-41eb-b335-1203dc85faaa-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.627970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.627976    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-home\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.628194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.627984    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a9ea32-a1a7-41eb-b335-1203dc85faaa-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.628194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.627993    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9pm\" (UniqueName: \"kubernetes.io/projected/70a9ea32-a1a7-41eb-b335-1203dc85faaa-kube-api-access-bb9pm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.628194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.628001    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-tls-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.628194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:58.628009    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f-dshm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:59.401895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.401804    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"
Oct 31 00:31:59.401895 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.401842    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4" event={"ID":"70a9ea32-a1a7-41eb-b335-1203dc85faaa","Type":"ContainerDied","Data":"f86c846875cc7d87ea36ad5dd7357e035f000e4e883f74baf6468bf84ad693b2"}
Oct 31 00:31:59.402493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.401901    5078 scope.go:122] "RemoveContainer" containerID="19f4559f3fe68403e292c5c93282060e525effc83e79d616d4672e16929314d7"
Oct 31 00:31:59.403498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.403439    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n" event={"ID":"0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f","Type":"ContainerDied","Data":"9e6e8797ee9705e2786f658fe587a6212f5e800cb02ca5dc08e50e027f11d7b2"}
Oct 31 00:31:59.403636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.403569    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"
Oct 31 00:31:59.414831 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.414801    5078 scope.go:122] "RemoveContainer" containerID="2ceb9a86170ccb669d35be64fbdd04adf0494932305afc3a62fda264aea95901"
Oct 31 00:31:59.438989 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.438962    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"]
Oct 31 00:31:59.445081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.445060    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/premium-w5-7b2341-kserve-bb7676bf4-2xzq4"]
Oct 31 00:31:59.458032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.458006    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"]
Oct 31 00:31:59.471978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:31:59.471957    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/simulator-w5-7b2341-kserve-c87fb5b59-svr9n"]
Oct 31 00:32:00.194878 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:00.194840    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" path="/var/lib/kubelet/pods/0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f/volumes"
Oct 31 00:32:00.195324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:00.195308    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" path="/var/lib/kubelet/pods/70a9ea32-a1a7-41eb-b335-1203dc85faaa/volumes"
Oct 31 00:32:04.461526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:04.461478    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"]
Oct 31 00:32:04.471891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:04.471853    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-7b2341-cbd5s"]
Oct 31 00:32:06.194261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:06.194228    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9" path="/var/lib/kubelet/pods/a6afdcd4-916a-43ba-a6d7-3ce4b1879ec9/volumes"
Oct 31 00:32:06.387412 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:32:06.387375    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:32:11.905858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:11.905818    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"]
Oct 31 00:32:11.906511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:11.906107    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" containerID="cri-o://947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e" gracePeriod=30
Oct 31 00:32:13.562340 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:13.562303    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:32:13.562784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:13.562369    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:32:14.672287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:14.672257    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-jrv29"]
Oct 31 00:32:14.672828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:14.672455    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager" containerID="cri-o://c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a" gracePeriod=30
Oct 31 00:32:14.935790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:14.935726    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:32:15.108275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.108241    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert\") pod \"68cb91e9-c6e4-460f-9318-49d79080dedb\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") "
Oct 31 00:32:15.108275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.108284    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs\") pod \"68cb91e9-c6e4-460f-9318-49d79080dedb\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") "
Oct 31 00:32:15.109114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.108322    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz\" (UniqueName: \"kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz\") pod \"68cb91e9-c6e4-460f-9318-49d79080dedb\" (UID: \"68cb91e9-c6e4-460f-9318-49d79080dedb\") "
Oct 31 00:32:15.110960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.110879    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs" pod "68cb91e9-c6e4-460f-9318-49d79080dedb" (UID: "68cb91e9-c6e4-460f-9318-49d79080dedb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:15.110960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.110917    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz" pod "68cb91e9-c6e4-460f-9318-49d79080dedb" (UID: "68cb91e9-c6e4-460f-9318-49d79080dedb"). InnerVolumeSpecName "kube-api-access-zfphz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:15.111161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.111049    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert" pod "68cb91e9-c6e4-460f-9318-49d79080dedb" (UID: "68cb91e9-c6e4-460f-9318-49d79080dedb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:15.210142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.210070    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfphz\" (UniqueName: \"kubernetes.io/projected/68cb91e9-c6e4-460f-9318-49d79080dedb-kube-api-access-zfphz\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.210142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.210095    5078 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-webhook-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.210142 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.210104    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cb91e9-c6e4-460f-9318-49d79080dedb-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.529992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.529904    5078 generic.go:356] "Generic (PLEG): container finished" podID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerID="c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a" exitCode=0
Oct 31 00:32:15.529992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.529973    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29"
Oct 31 00:32:15.530250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.529993    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" event={"ID":"68cb91e9-c6e4-460f-9318-49d79080dedb","Type":"ContainerDied","Data":"c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a"}
Oct 31 00:32:15.530250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.530070    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-jrv29" event={"ID":"68cb91e9-c6e4-460f-9318-49d79080dedb","Type":"ContainerDied","Data":"b2c810beb59dffec8927ce2cf7315c7e1075c61918a13449253a876196bfacbe"}
Oct 31 00:32:15.530250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.530100    5078 scope.go:122] "RemoveContainer" containerID="c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a"
Oct 31 00:32:15.542345 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.542330    5078 scope.go:122] "RemoveContainer" containerID="27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"
Oct 31 00:32:15.554332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.554313    5078 scope.go:122] "RemoveContainer" containerID="c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a"
Oct 31 00:32:15.554653 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:32:15.554625    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a\": container with ID starting with c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a not found: ID does not exist" containerID="c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a"
Oct 31 00:32:15.554731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.554661    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a"} err="failed to get container status \"c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a\": rpc error: code = NotFound desc = could not find container \"c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a\": container with ID starting with c81bd7243c4ec9e774e1eb05ba475fb15fcd2d93d300d7cca1e18cd64cfda32a not found: ID does not exist"
Oct 31 00:32:15.554731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.554678    5078 scope.go:122] "RemoveContainer" containerID="27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"
Oct 31 00:32:15.554928 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:32:15.554911    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e\": container with ID starting with 27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e not found: ID does not exist" containerID="27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"
Oct 31 00:32:15.554972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.554932    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e"} err="failed to get container status \"27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e\": rpc error: code = NotFound desc = could not find container \"27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e\": container with ID starting with 27ec48dba1468f589c7361e836d8972d74233b049ec896f5b3936cea21baf44e not found: ID does not exist"
Oct 31 00:32:15.561446 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.561425    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-jrv29"]
Oct 31 00:32:15.567905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:15.567887    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-jrv29"]
Oct 31 00:32:16.198223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:16.198184    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" path="/var/lib/kubelet/pods/68cb91e9-c6e4-460f-9318-49d79080dedb/volumes"
Oct 31 00:32:17.555545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.554736    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:32:17.562741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.562711    5078 generic.go:356] "Generic (PLEG): container finished" podID="d398e49a-c635-4945-8971-067877a21aea" containerID="947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e" exitCode=0
Oct 31 00:32:17.562890 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.562865    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" event={"ID":"d398e49a-c635-4945-8971-067877a21aea","Type":"ContainerDied","Data":"947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e"}
Oct 31 00:32:17.562937 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.562929    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv" event={"ID":"d398e49a-c635-4945-8971-067877a21aea","Type":"ContainerDied","Data":"0d05991466c326f40558784f2fc6f6248e80cc7b35b59100e73c7714cffc6475"}
Oct 31 00:32:17.562974 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.562948    5078 scope.go:122] "RemoveContainer" containerID="947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e"
Oct 31 00:32:17.563057 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.563046    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"
Oct 31 00:32:17.649502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.648715    5078 scope.go:122] "RemoveContainer" containerID="947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e"
Oct 31 00:32:17.649502 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:32:17.649082    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e\": container with ID starting with 947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e not found: ID does not exist" containerID="947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e"
Oct 31 00:32:17.649502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.649119    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e"} err="failed to get container status \"947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e\": rpc error: code = NotFound desc = could not find container \"947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e\": container with ID starting with 947ec4f65a16761dfd891614cb09bdf74e5bc70efe7ecef6cf11f1ff7bdfda9e not found: ID does not exist"
Oct 31 00:32:17.661635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661593    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.661786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661720    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.661858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661790    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.661858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661835    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.662000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661877    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.662000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661920    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.662000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661950    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.662000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.661980    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.662245 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.662021    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data\") pod \"d398e49a-c635-4945-8971-067877a21aea\" (UID: \"d398e49a-c635-4945-8971-067877a21aea\") "
Oct 31 00:32:17.663064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.662902    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:17.663198 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.663176    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:17.663444 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.663253    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:17.663665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.663605    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:17.663996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.663974    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:17.665064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.665030    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:32:17.665328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.665303    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "kube-api-access-ffl2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:17.666000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.665977    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:17.666483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.666437    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token" pod "d398e49a-c635-4945-8971-067877a21aea" (UID: "d398e49a-c635-4945-8971-067877a21aea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763568    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763601    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffl2p\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-kube-api-access-ffl2p\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763614    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763622    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763631    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d398e49a-c635-4945-8971-067877a21aea-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763641    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d398e49a-c635-4945-8971-067877a21aea-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763649    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.763652 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763657    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d398e49a-c635-4945-8971-067877a21aea-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.764040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.763666    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d398e49a-c635-4945-8971-067877a21aea-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:17.930161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.930123    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"]
Oct 31 00:32:17.944498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:17.943561    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-lg8hv"]
Oct 31 00:32:18.239487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:18.236252    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d398e49a-c635-4945-8971-067877a21aea" path="/var/lib/kubelet/pods/d398e49a-c635-4945-8971-067877a21aea/volumes"
Oct 31 00:32:31.552403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:31.552353    5078 scope.go:122] "RemoveContainer" containerID="88bd3643ec73affaa97ba32796139ecae97b8320469bbd592eb7930c79d3ac38"
Oct 31 00:32:31.565060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:31.565042    5078 scope.go:122] "RemoveContainer" containerID="220dc17a9beadfbc1b81321b4b577cc377de699f94ea0e91f86f2609cc67a5f2"
Oct 31 00:32:31.574410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:31.574394    5078 scope.go:122] "RemoveContainer" containerID="11ab478d9195f5d55c810081d1ee7ddd493b8a33256b10de14ba390e7aac5cc5"
Oct 31 00:32:36.385415 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:32:36.385376    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:32:39.815263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815234    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"]
Oct 31 00:32:39.815685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815672    5078 cpu_manager.go:414] "containerMap: removing container" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager"
Oct 31 00:32:39.815733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815686    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager"
Oct 31 00:32:39.815733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815698    5078 cpu_manager.go:414] "containerMap: removing container" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main"
Oct 31 00:32:39.815733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815703    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815715    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815752    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815760    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815764    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815773    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815778    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815805    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3026d1d2-6db0-4b98-a40d-f0486afa9ef9" containerName="revoke-keys"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815813    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3026d1d2-6db0-4b98-a40d-f0486afa9ef9" containerName="revoke-keys"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815825    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815832    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815845    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy"
Oct 31 00:32:39.815847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815850    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815866    5078 cpu_manager.go:414] "containerMap: removing container" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815871    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815956    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d398e49a-c635-4945-8971-067877a21aea" containerName="istio-proxy"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815964    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da4edd2-f1d2-478b-9bf1-1d5f3a3afa9f" containerName="main"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.815971    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d43026d-495b-4d5c-b30c-4ee2f68bb8f4" containerName="istio-proxy"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.816017    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3026d1d2-6db0-4b98-a40d-f0486afa9ef9" containerName="revoke-keys"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.816024    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.816030    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd56ff0f-da7a-4de9-b2a0-125e471de6d2" containerName="istio-proxy"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.816036    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.816043    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cb91e9-c6e4-460f-9318-49d79080dedb" containerName="manager"
Oct 31 00:32:39.816270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.816050    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a9ea32-a1a7-41eb-b335-1203dc85faaa" containerName="main"
Oct 31 00:32:39.821353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.821335    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:39.830402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.830378    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:32:39.833589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.833571    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:32:39.835824 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.835804    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"]
Oct 31 00:32:39.913240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.913205    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:32:39.913404 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.913308    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:32:39.991859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.991828    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:39.992003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.991868    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:39.992003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:39.991987    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnqgd\" (UniqueName: \"kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.093216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.093134    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnqgd\" (UniqueName: \"kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.093359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.093218    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.093359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.093247    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.093855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.093831    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.095740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.095719    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.107109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.107090    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnqgd\" (UniqueName: \"kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd\") pod \"maas-api-revoke-sub-85237e45-h9wdq\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.135102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.135079    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:40.412449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.412421    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"]
Oct 31 00:32:40.414318 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:32:40.414291    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a39bad_e1f2_45a4_a710_6e346c67f942.slice/crio-6b11ecd6e8cbe5b55873e6c9eaaacb6faee3cbb400a3e4faea2f9d1f9af28c6f WatchSource:0}: Error finding container 6b11ecd6e8cbe5b55873e6c9eaaacb6faee3cbb400a3e4faea2f9d1f9af28c6f: Status 404 returned error can't find the container with id 6b11ecd6e8cbe5b55873e6c9eaaacb6faee3cbb400a3e4faea2f9d1f9af28c6f
Oct 31 00:32:40.713382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.713353    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq" event={"ID":"f4a39bad-e1f2-45a4-a710-6e346c67f942","Type":"ContainerStarted","Data":"3d414de2984395d6cee9151ae8aed3bc78f2c5a1000115c62afa8cacbbc04e6f"}
Oct 31 00:32:40.713540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.713411    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq" event={"ID":"f4a39bad-e1f2-45a4-a710-6e346c67f942","Type":"ContainerStarted","Data":"6b11ecd6e8cbe5b55873e6c9eaaacb6faee3cbb400a3e4faea2f9d1f9af28c6f"}
Oct 31 00:32:40.736938 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:40.736892    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq" podStartSLOduration=1.736877317 podStartE2EDuration="1.736877317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:40.734560485 +0000 UTC m=+2328.680931203" watchObservedRunningTime="2026-10-31 00:32:40.736877317 +0000 UTC m=+2328.683248031"
Oct 31 00:32:41.722002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:41.721971    5078 generic.go:356] "Generic (PLEG): container finished" podID="f4a39bad-e1f2-45a4-a710-6e346c67f942" containerID="3d414de2984395d6cee9151ae8aed3bc78f2c5a1000115c62afa8cacbbc04e6f" exitCode=0
Oct 31 00:32:41.722435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:41.722063    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq" event={"ID":"f4a39bad-e1f2-45a4-a710-6e346c67f942","Type":"ContainerDied","Data":"3d414de2984395d6cee9151ae8aed3bc78f2c5a1000115c62afa8cacbbc04e6f"}
Oct 31 00:32:42.873897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:42.873873    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:43.025220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.025140    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd\" (UniqueName: \"kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd\") pod \"f4a39bad-e1f2-45a4-a710-6e346c67f942\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") "
Oct 31 00:32:43.025220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.025195    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca\") pod \"f4a39bad-e1f2-45a4-a710-6e346c67f942\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") "
Oct 31 00:32:43.025537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.025227    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca\") pod \"f4a39bad-e1f2-45a4-a710-6e346c67f942\" (UID: \"f4a39bad-e1f2-45a4-a710-6e346c67f942\") "
Oct 31 00:32:43.025707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.025685    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca" pod "f4a39bad-e1f2-45a4-a710-6e346c67f942" (UID: "f4a39bad-e1f2-45a4-a710-6e346c67f942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:43.027538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.027517    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca" pod "f4a39bad-e1f2-45a4-a710-6e346c67f942" (UID: "f4a39bad-e1f2-45a4-a710-6e346c67f942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:43.027767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.027555    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd" pod "f4a39bad-e1f2-45a4-a710-6e346c67f942" (UID: "f4a39bad-e1f2-45a4-a710-6e346c67f942"). InnerVolumeSpecName "kube-api-access-nnqgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:43.126400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.126372    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnqgd\" (UniqueName: \"kubernetes.io/projected/f4a39bad-e1f2-45a4-a710-6e346c67f942-kube-api-access-nnqgd\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.126400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.126396    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.126400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.126407    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a39bad-e1f2-45a4-a710-6e346c67f942-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.735815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.735789    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"
Oct 31 00:32:43.736004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.735828    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq" event={"ID":"f4a39bad-e1f2-45a4-a710-6e346c67f942","Type":"ContainerDied","Data":"6b11ecd6e8cbe5b55873e6c9eaaacb6faee3cbb400a3e4faea2f9d1f9af28c6f"}
Oct 31 00:32:43.736004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:32:43.735879    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b11ecd6e8cbe5b55873e6c9eaaacb6faee3cbb400a3e4faea2f9d1f9af28c6f"
Oct 31 00:33:06.385809 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:33:06.385767    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:33:31.657672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:31.657594    5078 scope.go:122] "RemoveContainer" containerID="9f900d76591805fb4458f5052a9906e3b1b60d1983c442380e3d516fed527053"
Oct 31 00:33:36.082528 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.082488    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"]
Oct 31 00:33:36.083234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.083212    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f4a39bad-e1f2-45a4-a710-6e346c67f942" containerName="revoke-keys"
Oct 31 00:33:36.083332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.083237    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a39bad-e1f2-45a4-a710-6e346c67f942" containerName="revoke-keys"
Oct 31 00:33:36.083390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.083358    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a39bad-e1f2-45a4-a710-6e346c67f942" containerName="revoke-keys"
Oct 31 00:33:36.088048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.088024    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.092988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.092962    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:33:36.092988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.092969    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:33:36.093169 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.093064    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:33:36.093386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.093364    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:33:36.096441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.096415    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"]
Oct 31 00:33:36.189930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.189892    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.190126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.189973    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdkk\" (UniqueName: \"kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.190186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.190163    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.291746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.291713    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.291924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.291815    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.291924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.291882    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdkk\" (UniqueName: \"kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.292451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.292431    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.294552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.294529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.305367 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.305337    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdkk\" (UniqueName: \"kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk\") pod \"maas-api-revoke-sub-c1e271a9-2qr8f\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.385552 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:33:36.385519    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:33:36.413064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.413033    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:36.691870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:36.691840    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"]
Oct 31 00:33:36.693266 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:33:36.693239    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bed518f_78fb_48c9_ab10_2577679aa963.slice/crio-9acf8f3942d2a79529f08d9c0d16dcd91aeb2e4612af0f3276739ba281120842 WatchSource:0}: Error finding container 9acf8f3942d2a79529f08d9c0d16dcd91aeb2e4612af0f3276739ba281120842: Status 404 returned error can't find the container with id 9acf8f3942d2a79529f08d9c0d16dcd91aeb2e4612af0f3276739ba281120842
Oct 31 00:33:37.107881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:37.107847    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f" event={"ID":"4bed518f-78fb-48c9-ab10-2577679aa963","Type":"ContainerStarted","Data":"17eb6ae3e71ef723c4df7fe7d4fd62ab9209afd33baaf9db8384d3b9d23f69dd"}
Oct 31 00:33:37.108285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:37.107910    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f" event={"ID":"4bed518f-78fb-48c9-ab10-2577679aa963","Type":"ContainerStarted","Data":"9acf8f3942d2a79529f08d9c0d16dcd91aeb2e4612af0f3276739ba281120842"}
Oct 31 00:33:37.129966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:37.129919    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f" podStartSLOduration=1.1299062420000001 podStartE2EDuration="1.129906242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:37.12937939 +0000 UTC m=+2385.075750112" watchObservedRunningTime="2026-10-31 00:33:37.129906242 +0000 UTC m=+2385.076276960"
Oct 31 00:33:38.116329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:38.116237    5078 generic.go:356] "Generic (PLEG): container finished" podID="4bed518f-78fb-48c9-ab10-2577679aa963" containerID="17eb6ae3e71ef723c4df7fe7d4fd62ab9209afd33baaf9db8384d3b9d23f69dd" exitCode=0
Oct 31 00:33:38.116329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:38.116290    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f" event={"ID":"4bed518f-78fb-48c9-ab10-2577679aa963","Type":"ContainerDied","Data":"17eb6ae3e71ef723c4df7fe7d4fd62ab9209afd33baaf9db8384d3b9d23f69dd"}
Oct 31 00:33:39.268328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.268305    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:39.324808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.324776    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca\") pod \"4bed518f-78fb-48c9-ab10-2577679aa963\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") "
Oct 31 00:33:39.324952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.324920    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca\") pod \"4bed518f-78fb-48c9-ab10-2577679aa963\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") "
Oct 31 00:33:39.325001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.324967    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk\" (UniqueName: \"kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk\") pod \"4bed518f-78fb-48c9-ab10-2577679aa963\" (UID: \"4bed518f-78fb-48c9-ab10-2577679aa963\") "
Oct 31 00:33:39.325320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.325291    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca" pod "4bed518f-78fb-48c9-ab10-2577679aa963" (UID: "4bed518f-78fb-48c9-ab10-2577679aa963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:33:39.327217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.327195    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca" pod "4bed518f-78fb-48c9-ab10-2577679aa963" (UID: "4bed518f-78fb-48c9-ab10-2577679aa963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:33:39.327324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.327245    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk" pod "4bed518f-78fb-48c9-ab10-2577679aa963" (UID: "4bed518f-78fb-48c9-ab10-2577679aa963"). InnerVolumeSpecName "kube-api-access-wgdkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:39.426530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.426506    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:39.426530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.426528    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdkk\" (UniqueName: \"kubernetes.io/projected/4bed518f-78fb-48c9-ab10-2577679aa963-kube-api-access-wgdkk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:39.426692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:39.426538    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bed518f-78fb-48c9-ab10-2577679aa963-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:40.130000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:40.129971    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"
Oct 31 00:33:40.130165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:40.130000    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f" event={"ID":"4bed518f-78fb-48c9-ab10-2577679aa963","Type":"ContainerDied","Data":"9acf8f3942d2a79529f08d9c0d16dcd91aeb2e4612af0f3276739ba281120842"}
Oct 31 00:33:40.130165 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:40.130054    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9acf8f3942d2a79529f08d9c0d16dcd91aeb2e4612af0f3276739ba281120842"
Oct 31 00:33:54.493247 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.493219    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:33:54.493766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.493745    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:33:54.498851 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.498830    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:33:54.499766 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.499750    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:33:54.881160 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.881085    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"]
Oct 31 00:33:54.881596 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.881582    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4bed518f-78fb-48c9-ab10-2577679aa963" containerName="revoke-keys"
Oct 31 00:33:54.881647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.881597    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bed518f-78fb-48c9-ab10-2577679aa963" containerName="revoke-keys"
Oct 31 00:33:54.881688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.881676    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bed518f-78fb-48c9-ab10-2577679aa963" containerName="revoke-keys"
Oct 31 00:33:54.885046 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.885020    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:54.894272 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.894246    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:33:54.894435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.894278    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:33:54.894435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.894297    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:33:54.894435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.894363    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:33:54.896032 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.895990    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"]
Oct 31 00:33:54.996925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.996894    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:54.997100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.996944    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnwc7\" (UniqueName: \"kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:54.997100 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:54.997056    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.098014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.097986    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwc7\" (UniqueName: \"kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.098204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.098068    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.098204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.098169    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.098755 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.098734    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.100701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.100680    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.112161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.112140    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwc7\" (UniqueName: \"kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7\") pod \"maas-api-revoke-sub-2e1f5b5f-px52p\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.202344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.202323    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:55.477061 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:55.477033    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"]
Oct 31 00:33:55.477574 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:33:55.477550    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcef99b_70f8_47ee_94cb_aeeef9cf339f.slice/crio-e37c699d49fea04182b5150280eed6713225a0aeac67e20ad2e22af66955a8e9 WatchSource:0}: Error finding container e37c699d49fea04182b5150280eed6713225a0aeac67e20ad2e22af66955a8e9: Status 404 returned error can't find the container with id e37c699d49fea04182b5150280eed6713225a0aeac67e20ad2e22af66955a8e9
Oct 31 00:33:56.276538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:56.276500    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p" event={"ID":"efcef99b-70f8-47ee-94cb-aeeef9cf339f","Type":"ContainerStarted","Data":"22ff7b0dfbb5b6890692ef38562807d1b0918122fa339f6510f06f4a4005a17e"}
Oct 31 00:33:56.276921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:56.276561    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p" event={"ID":"efcef99b-70f8-47ee-94cb-aeeef9cf339f","Type":"ContainerStarted","Data":"e37c699d49fea04182b5150280eed6713225a0aeac67e20ad2e22af66955a8e9"}
Oct 31 00:33:56.300143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:56.300091    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p" podStartSLOduration=2.300076471 podStartE2EDuration="2.300076471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:56.296504452 +0000 UTC m=+2404.242875193" watchObservedRunningTime="2026-10-31 00:33:56.300076471 +0000 UTC m=+2404.246447188"
Oct 31 00:33:57.283925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:57.283882    5078 generic.go:356] "Generic (PLEG): container finished" podID="efcef99b-70f8-47ee-94cb-aeeef9cf339f" containerID="22ff7b0dfbb5b6890692ef38562807d1b0918122fa339f6510f06f4a4005a17e" exitCode=0
Oct 31 00:33:57.283925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:57.283928    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p" event={"ID":"efcef99b-70f8-47ee-94cb-aeeef9cf339f","Type":"ContainerDied","Data":"22ff7b0dfbb5b6890692ef38562807d1b0918122fa339f6510f06f4a4005a17e"}
Oct 31 00:33:58.438902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.438875    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:58.531954 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.531923    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca\") pod \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") "
Oct 31 00:33:58.532118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.531984    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7\" (UniqueName: \"kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7\") pod \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") "
Oct 31 00:33:58.532118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.532022    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca\") pod \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\" (UID: \"efcef99b-70f8-47ee-94cb-aeeef9cf339f\") "
Oct 31 00:33:58.532449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.532425    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca" pod "efcef99b-70f8-47ee-94cb-aeeef9cf339f" (UID: "efcef99b-70f8-47ee-94cb-aeeef9cf339f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:33:58.534190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.534161    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca" pod "efcef99b-70f8-47ee-94cb-aeeef9cf339f" (UID: "efcef99b-70f8-47ee-94cb-aeeef9cf339f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:33:58.534284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.534242    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7" pod "efcef99b-70f8-47ee-94cb-aeeef9cf339f" (UID: "efcef99b-70f8-47ee-94cb-aeeef9cf339f"). InnerVolumeSpecName "kube-api-access-wnwc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:58.633038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.633004    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:58.633038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.633038    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efcef99b-70f8-47ee-94cb-aeeef9cf339f-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:58.633038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:58.633048    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnwc7\" (UniqueName: \"kubernetes.io/projected/efcef99b-70f8-47ee-94cb-aeeef9cf339f-kube-api-access-wnwc7\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:59.296713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:59.296685    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"
Oct 31 00:33:59.296871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:59.296727    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p" event={"ID":"efcef99b-70f8-47ee-94cb-aeeef9cf339f","Type":"ContainerDied","Data":"e37c699d49fea04182b5150280eed6713225a0aeac67e20ad2e22af66955a8e9"}
Oct 31 00:33:59.296871 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:33:59.296776    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37c699d49fea04182b5150280eed6713225a0aeac67e20ad2e22af66955a8e9"
Oct 31 00:34:05.979284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.979245    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"]
Oct 31 00:34:05.979913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.979811    5078 cpu_manager.go:414] "containerMap: removing container" podUID="efcef99b-70f8-47ee-94cb-aeeef9cf339f" containerName="revoke-keys"
Oct 31 00:34:05.979913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.979827    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efcef99b-70f8-47ee-94cb-aeeef9cf339f" containerName="revoke-keys"
Oct 31 00:34:05.980052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.979958    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="efcef99b-70f8-47ee-94cb-aeeef9cf339f" containerName="revoke-keys"
Oct 31 00:34:05.983411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.983390    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:05.992491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.992448    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:34:05.992782 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.992761    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:34:05.992984 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.992967    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:34:05.993341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.992772    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:34:05.993719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:05.993515    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"]
Oct 31 00:34:06.108251 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.108211    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.108482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.108326    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.108482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.108400    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnvd\" (UniqueName: \"kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.209837 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.209808    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnvd\" (UniqueName: \"kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.210042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.209914    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.210042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.209941    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.210580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.210557    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.212506 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.212455    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.226239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.226219    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnvd\" (UniqueName: \"kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd\") pod \"maas-api-revoke-sub-ec843018-d6wpz\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.304941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.304847    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:06.409973 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:34:06.409885    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:34:06.606608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:06.606573    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"]
Oct 31 00:34:06.607098 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:34:06.607072    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa8dce9_848b_437b_8d9e_e146549d6ef6.slice/crio-3b4fa03ae2cb1aea3380a6edabf136b0b515a74ccb48cb9fdcfc79554277d3a0 WatchSource:0}: Error finding container 3b4fa03ae2cb1aea3380a6edabf136b0b515a74ccb48cb9fdcfc79554277d3a0: Status 404 returned error can't find the container with id 3b4fa03ae2cb1aea3380a6edabf136b0b515a74ccb48cb9fdcfc79554277d3a0
Oct 31 00:34:07.359386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:07.359352    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz" event={"ID":"0aa8dce9-848b-437b-8d9e-e146549d6ef6","Type":"ContainerStarted","Data":"0d49ea8d52912847a656a0571432d3098800aa17b56fd9f63df1e21dea8713bd"}
Oct 31 00:34:07.359906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:07.359420    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz" event={"ID":"0aa8dce9-848b-437b-8d9e-e146549d6ef6","Type":"ContainerStarted","Data":"3b4fa03ae2cb1aea3380a6edabf136b0b515a74ccb48cb9fdcfc79554277d3a0"}
Oct 31 00:34:07.388101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:07.388044    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz" podStartSLOduration=2.388023926 podStartE2EDuration="2.388023926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:07.38317251 +0000 UTC m=+2415.329543229" watchObservedRunningTime="2026-10-31 00:34:07.388023926 +0000 UTC m=+2415.334394644"
Oct 31 00:34:08.368042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:08.368001    5078 generic.go:356] "Generic (PLEG): container finished" podID="0aa8dce9-848b-437b-8d9e-e146549d6ef6" containerID="0d49ea8d52912847a656a0571432d3098800aa17b56fd9f63df1e21dea8713bd" exitCode=0
Oct 31 00:34:08.368555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:08.368145    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz" event={"ID":"0aa8dce9-848b-437b-8d9e-e146549d6ef6","Type":"ContainerDied","Data":"0d49ea8d52912847a656a0571432d3098800aa17b56fd9f63df1e21dea8713bd"}
Oct 31 00:34:10.027708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.027679    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:10.132487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.118788    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd\" (UniqueName: \"kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd\") pod \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") "
Oct 31 00:34:10.132487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.118945    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca\") pod \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") "
Oct 31 00:34:10.132487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.119023    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca\") pod \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\" (UID: \"0aa8dce9-848b-437b-8d9e-e146549d6ef6\") "
Oct 31 00:34:10.132487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.127652    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca" pod "0aa8dce9-848b-437b-8d9e-e146549d6ef6" (UID: "0aa8dce9-848b-437b-8d9e-e146549d6ef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:10.162108 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.162072    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca" pod "0aa8dce9-848b-437b-8d9e-e146549d6ef6" (UID: "0aa8dce9-848b-437b-8d9e-e146549d6ef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:10.175695 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.175600    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd" pod "0aa8dce9-848b-437b-8d9e-e146549d6ef6" (UID: "0aa8dce9-848b-437b-8d9e-e146549d6ef6"). InnerVolumeSpecName "kube-api-access-4hnvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:10.244490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.237779    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:10.244490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.237816    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8dce9-848b-437b-8d9e-e146549d6ef6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:10.244490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.237834    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnvd\" (UniqueName: \"kubernetes.io/projected/0aa8dce9-848b-437b-8d9e-e146549d6ef6-kube-api-access-4hnvd\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:10.404789 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.404750    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz" event={"ID":"0aa8dce9-848b-437b-8d9e-e146549d6ef6","Type":"ContainerDied","Data":"3b4fa03ae2cb1aea3380a6edabf136b0b515a74ccb48cb9fdcfc79554277d3a0"}
Oct 31 00:34:10.405062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.405045    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b4fa03ae2cb1aea3380a6edabf136b0b515a74ccb48cb9fdcfc79554277d3a0"
Oct 31 00:34:10.405235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:10.405223    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"
Oct 31 00:34:31.680786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:31.680755    5078 scope.go:122] "RemoveContainer" containerID="f02dbb66a2c11d4f2e6245ffd23f46c3fd4bfd619e82bac7e1af6961aa065978"
Oct 31 00:34:31.690261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:31.690243    5078 scope.go:122] "RemoveContainer" containerID="54d802f2eec8497b1e5385bd52737ff42b39afdaefb7165ea6361c4a46b2f028"
Oct 31 00:34:31.698912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:31.698889    5078 scope.go:122] "RemoveContainer" containerID="df383ec9849a853e257de5780b5a4a10b2eb546ec85ac176a76bb15c54a2c1ca"
Oct 31 00:34:31.707790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:31.707773    5078 scope.go:122] "RemoveContainer" containerID="afad5a1c5eed461d98ea6f81c8580063234aa76e8510c795095781654a988615"
Oct 31 00:34:32.283331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.283298    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"]
Oct 31 00:34:32.283765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.283748    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0aa8dce9-848b-437b-8d9e-e146549d6ef6" containerName="revoke-keys"
Oct 31 00:34:32.283765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.283764    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa8dce9-848b-437b-8d9e-e146549d6ef6" containerName="revoke-keys"
Oct 31 00:34:32.283965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.283859    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa8dce9-848b-437b-8d9e-e146549d6ef6" containerName="revoke-keys"
Oct 31 00:34:32.288280 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.288259    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.294449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.294423    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:34:32.295432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.295406    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:34:32.296375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.296354    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"]
Oct 31 00:34:32.365702 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.365674    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:34:32.365872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.365730    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:34:32.379283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.379259    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ht4\" (UniqueName: \"kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.379380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.379298    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.379449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.379397    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.480750 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.480708    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ht4\" (UniqueName: \"kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.480947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.480759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.480947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.480810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.481503 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.481443    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.483410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.483386    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.493983 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.493964    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ht4\" (UniqueName: \"kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4\") pod \"maas-api-revoke-sub-046e1343-b7lxj\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.604788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.604732    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:32.885133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:32.885104    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"]
Oct 31 00:34:32.886015 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:34:32.885987    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d2694c_c73e_45cb_86c8_71cb366ea06f.slice/crio-e9d893108c7dbb898e7a3a350209c3d8ce64fedbb84a3259b0e3a3190ecce1c8 WatchSource:0}: Error finding container e9d893108c7dbb898e7a3a350209c3d8ce64fedbb84a3259b0e3a3190ecce1c8: Status 404 returned error can't find the container with id e9d893108c7dbb898e7a3a350209c3d8ce64fedbb84a3259b0e3a3190ecce1c8
Oct 31 00:34:33.593174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:33.593091    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj" event={"ID":"74d2694c-c73e-45cb-86c8-71cb366ea06f","Type":"ContainerStarted","Data":"c675eca1fef963aa3cf75f23271a12ec315ab3011727bce22fdff1428fda52e7"}
Oct 31 00:34:33.593174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:33.593159    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj" event={"ID":"74d2694c-c73e-45cb-86c8-71cb366ea06f","Type":"ContainerStarted","Data":"e9d893108c7dbb898e7a3a350209c3d8ce64fedbb84a3259b0e3a3190ecce1c8"}
Oct 31 00:34:33.616346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:33.616302    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj" podStartSLOduration=1.6162897489999999 podStartE2EDuration="1.616289749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:33.612410259 +0000 UTC m=+2441.558780977" watchObservedRunningTime="2026-10-31 00:34:33.616289749 +0000 UTC m=+2441.562660466"
Oct 31 00:34:34.600837 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:34.600802    5078 generic.go:356] "Generic (PLEG): container finished" podID="74d2694c-c73e-45cb-86c8-71cb366ea06f" containerID="c675eca1fef963aa3cf75f23271a12ec315ab3011727bce22fdff1428fda52e7" exitCode=0
Oct 31 00:34:34.601261 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:34.600900    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj" event={"ID":"74d2694c-c73e-45cb-86c8-71cb366ea06f","Type":"ContainerDied","Data":"c675eca1fef963aa3cf75f23271a12ec315ab3011727bce22fdff1428fda52e7"}
Oct 31 00:34:35.768725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.768700    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:35.913427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.913403    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca\") pod \"74d2694c-c73e-45cb-86c8-71cb366ea06f\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") "
Oct 31 00:34:35.913618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.913443    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4\" (UniqueName: \"kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4\") pod \"74d2694c-c73e-45cb-86c8-71cb366ea06f\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") "
Oct 31 00:34:35.913618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.913565    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca\") pod \"74d2694c-c73e-45cb-86c8-71cb366ea06f\" (UID: \"74d2694c-c73e-45cb-86c8-71cb366ea06f\") "
Oct 31 00:34:35.913829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.913807    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca" pod "74d2694c-c73e-45cb-86c8-71cb366ea06f" (UID: "74d2694c-c73e-45cb-86c8-71cb366ea06f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:35.915822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.915792    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca" pod "74d2694c-c73e-45cb-86c8-71cb366ea06f" (UID: "74d2694c-c73e-45cb-86c8-71cb366ea06f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:35.915959 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:35.915944    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4" pod "74d2694c-c73e-45cb-86c8-71cb366ea06f" (UID: "74d2694c-c73e-45cb-86c8-71cb366ea06f"). InnerVolumeSpecName "kube-api-access-j8ht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:36.014784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:36.014752    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:36.014784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:36.014779    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ht4\" (UniqueName: \"kubernetes.io/projected/74d2694c-c73e-45cb-86c8-71cb366ea06f-kube-api-access-j8ht4\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:36.014784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:36.014791    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d2694c-c73e-45cb-86c8-71cb366ea06f-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:36.385491 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:34:36.385439    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:34:36.614786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:36.614708    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"
Oct 31 00:34:36.614786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:36.614746    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj" event={"ID":"74d2694c-c73e-45cb-86c8-71cb366ea06f","Type":"ContainerDied","Data":"e9d893108c7dbb898e7a3a350209c3d8ce64fedbb84a3259b0e3a3190ecce1c8"}
Oct 31 00:34:36.614985 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:36.614790    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d893108c7dbb898e7a3a350209c3d8ce64fedbb84a3259b0e3a3190ecce1c8"
Oct 31 00:34:42.575313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.575277    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"]
Oct 31 00:34:42.575850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.575802    5078 cpu_manager.go:414] "containerMap: removing container" podUID="74d2694c-c73e-45cb-86c8-71cb366ea06f" containerName="revoke-keys"
Oct 31 00:34:42.575850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.575816    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d2694c-c73e-45cb-86c8-71cb366ea06f" containerName="revoke-keys"
Oct 31 00:34:42.575946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.575921    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d2694c-c73e-45cb-86c8-71cb366ea06f" containerName="revoke-keys"
Oct 31 00:34:42.579746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.579725    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.603252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.603231    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:34:42.603406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.603263    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:34:42.603406 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.603371    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:34:42.603564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.603530    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:34:42.617182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.617144    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"]
Oct 31 00:34:42.676257 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.676221    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.676434 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.676361    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4tk\" (UniqueName: \"kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.676534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.676524    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.777540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.777512    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.777729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.777549    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.777729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.777601    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4tk\" (UniqueName: \"kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.778300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.778274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.779986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.779967    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.791778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.791751    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4tk\" (UniqueName: \"kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk\") pod \"maas-api-revoke-sub-d3fba056-m9drm\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:42.894555 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:42.894530    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:43.177403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:43.177373    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"]
Oct 31 00:34:43.178756 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:34:43.178726    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba0d5d6_c8e0_4fed_a4fb_144ece03c5d5.slice/crio-be6d81274e658a433ea2eaa3f8ccea1f544e71d690184fe318256c048e32967a WatchSource:0}: Error finding container be6d81274e658a433ea2eaa3f8ccea1f544e71d690184fe318256c048e32967a: Status 404 returned error can't find the container with id be6d81274e658a433ea2eaa3f8ccea1f544e71d690184fe318256c048e32967a
Oct 31 00:34:43.659293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:43.659258    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm" event={"ID":"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5","Type":"ContainerStarted","Data":"aef0ada785924970d16b7eadb19d04ad0d15b819117ccefecb3e784092f178d8"}
Oct 31 00:34:43.659811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:43.659328    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm" event={"ID":"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5","Type":"ContainerStarted","Data":"be6d81274e658a433ea2eaa3f8ccea1f544e71d690184fe318256c048e32967a"}
Oct 31 00:34:43.684070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:43.684025    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm" podStartSLOduration=1.684011116 podStartE2EDuration="1.684011116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:43.680505971 +0000 UTC m=+2451.626876689" watchObservedRunningTime="2026-10-31 00:34:43.684011116 +0000 UTC m=+2451.630381863"
Oct 31 00:34:44.629540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:44.620346    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"]
Oct 31 00:34:44.629540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:44.620931    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery" containerID="cri-o://ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8" gracePeriod=30
Oct 31 00:34:44.719536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:44.718905    5078 generic.go:356] "Generic (PLEG): container finished" podID="4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" containerID="aef0ada785924970d16b7eadb19d04ad0d15b819117ccefecb3e784092f178d8" exitCode=0
Oct 31 00:34:44.719536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:44.719131    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm" event={"ID":"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5","Type":"ContainerDied","Data":"aef0ada785924970d16b7eadb19d04ad0d15b819117ccefecb3e784092f178d8"}
Oct 31 00:34:45.556485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.553711    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.677778    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.677852    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.677990    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.678030    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.678080    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.678149    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.678488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.678183    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap\") pod \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\" (UID: \"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8\") "
Oct 31 00:34:45.725756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.724615    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "kube-api-access-s86pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:45.736518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.724683    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:45.755497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.753514    5078 generic.go:356] "Generic (PLEG): container finished" podID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerID="ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8" exitCode=0
Oct 31 00:34:45.755497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.753922    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"
Oct 31 00:34:45.755497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.754274    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" event={"ID":"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8","Type":"ContainerDied","Data":"ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8"}
Oct 31 00:34:45.755497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.754318    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2" event={"ID":"3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8","Type":"ContainerDied","Data":"fe0c4db43c69c080f77e754ff025c5e77a9d6df5593b59f42ca263c083f7b390"}
Oct 31 00:34:45.755497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.754343    5078 scope.go:122] "RemoveContainer" containerID="ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8"
Oct 31 00:34:45.785486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.779758    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s86pm\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-kube-api-access-s86pm\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.785486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.779788    5078 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-ca-configmap\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.814162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.811813    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:45.814162 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.813558    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:45.837143 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.836615    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:45.837368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.837165    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:45.861935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.861605    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token" pod "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" (UID: "3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:45.884330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.884201    5078 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-csr-dns-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.884330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.884243    5078 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-cacerts\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.884330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.884261    5078 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-kubeconfig\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.884330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.884277    5078 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-local-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.884330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.884292    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:45.928041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.927891    5078 scope.go:122] "RemoveContainer" containerID="ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8"
Oct 31 00:34:45.942764 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:34:45.942649    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8\": container with ID starting with ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8 not found: ID does not exist" containerID="ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8"
Oct 31 00:34:45.942764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:45.942695    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8"} err="failed to get container status \"ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8\": rpc error: code = NotFound desc = could not find container \"ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8\": container with ID starting with ec056d456a691e1ff434eea8ec6774240d85aed281ed3343f1d6299ea2eb91a8 not found: ID does not exist"
Oct 31 00:34:46.159574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.148270    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"]
Oct 31 00:34:46.159574 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.148322    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qnjc2"]
Oct 31 00:34:46.232879 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.232316    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" path="/var/lib/kubelet/pods/3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8/volumes"
Oct 31 00:34:46.261951 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.261919    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:46.421925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.421891    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca\") pod \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") "
Oct 31 00:34:46.421925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.421929    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca\") pod \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") "
Oct 31 00:34:46.422205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.421980    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk\" (UniqueName: \"kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk\") pod \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\" (UID: \"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5\") "
Oct 31 00:34:46.422360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.422335    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca" pod "4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" (UID: "4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:46.424491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.424440    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca" pod "4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" (UID: "4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:46.424642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.424629    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk" pod "4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" (UID: "4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5"). InnerVolumeSpecName "kube-api-access-8p4tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:46.523592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.523555    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:46.523592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.523591    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:46.523846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.523604    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4tk\" (UniqueName: \"kubernetes.io/projected/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5-kube-api-access-8p4tk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:46.762352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.762263    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"
Oct 31 00:34:46.762352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.762305    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm" event={"ID":"4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5","Type":"ContainerDied","Data":"be6d81274e658a433ea2eaa3f8ccea1f544e71d690184fe318256c048e32967a"}
Oct 31 00:34:46.762912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:34:46.762359    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be6d81274e658a433ea2eaa3f8ccea1f544e71d690184fe318256c048e32967a"
Oct 31 00:35:03.740308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740216    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"]
Oct 31 00:35:03.740761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740745    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" containerName="revoke-keys"
Oct 31 00:35:03.740803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740762    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" containerName="revoke-keys"
Oct 31 00:35:03.740803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740784    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery"
Oct 31 00:35:03.740803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740789    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery"
Oct 31 00:35:03.740928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740893    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" containerName="revoke-keys"
Oct 31 00:35:03.740928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.740910    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb8a6fe-dc9f-436e-9bbc-9f52c727f5b8" containerName="discovery"
Oct 31 00:35:03.744655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.744634    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.768183 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.768152    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"]
Oct 31 00:35:03.768908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.768888    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:35:03.769042 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.768959    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:35:03.770944 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.770918    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:35:03.770944 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.770928    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:35:03.829519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.829449    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.829728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.829601    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.829728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.829639    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldff\" (UniqueName: \"kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.930826 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.930794    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.931025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.930869    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.931025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.930907    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldff\" (UniqueName: \"kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.931629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.931605    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.933347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.933327    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:03.943965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:03.943942    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldff\" (UniqueName: \"kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff\") pod \"maas-api-revoke-sub-ae594979-gxpgz\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:04.060793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:04.060713    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:04.337535 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:04.337502    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"]
Oct 31 00:35:04.338161 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:35:04.338136    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fdafa5_dc9b_4911_9f82_a14ce4c57494.slice/crio-23f9ff3797cb579d4bfcbc35c4758223935222b15b5267a42ccf9b0f82a3d2ee WatchSource:0}: Error finding container 23f9ff3797cb579d4bfcbc35c4758223935222b15b5267a42ccf9b0f82a3d2ee: Status 404 returned error can't find the container with id 23f9ff3797cb579d4bfcbc35c4758223935222b15b5267a42ccf9b0f82a3d2ee
Oct 31 00:35:04.904646 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:04.904604    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz" event={"ID":"90fdafa5-dc9b-4911-9f82-a14ce4c57494","Type":"ContainerStarted","Data":"e9f4e882764cfc7655d90e830b60befebad6338d072157b8f435e508868e7675"}
Oct 31 00:35:04.905040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:04.904663    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz" event={"ID":"90fdafa5-dc9b-4911-9f82-a14ce4c57494","Type":"ContainerStarted","Data":"23f9ff3797cb579d4bfcbc35c4758223935222b15b5267a42ccf9b0f82a3d2ee"}
Oct 31 00:35:04.926752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:04.926707    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz" podStartSLOduration=1.926694485 podStartE2EDuration="1.926694485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:04.925220145 +0000 UTC m=+2472.871590862" watchObservedRunningTime="2026-10-31 00:35:04.926694485 +0000 UTC m=+2472.873065201"
Oct 31 00:35:05.912288 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:05.912248    5078 generic.go:356] "Generic (PLEG): container finished" podID="90fdafa5-dc9b-4911-9f82-a14ce4c57494" containerID="e9f4e882764cfc7655d90e830b60befebad6338d072157b8f435e508868e7675" exitCode=0
Oct 31 00:35:05.912729 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:05.912315    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz" event={"ID":"90fdafa5-dc9b-4911-9f82-a14ce4c57494","Type":"ContainerDied","Data":"e9f4e882764cfc7655d90e830b60befebad6338d072157b8f435e508868e7675"}
Oct 31 00:35:06.385689 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:35:06.385653    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:35:07.067801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.067776    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:07.163611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.163583    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff\" (UniqueName: \"kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff\") pod \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") "
Oct 31 00:35:07.163807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.163640    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca\") pod \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") "
Oct 31 00:35:07.163807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.163780    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca\") pod \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\" (UID: \"90fdafa5-dc9b-4911-9f82-a14ce4c57494\") "
Oct 31 00:35:07.164114 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.164091    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca" pod "90fdafa5-dc9b-4911-9f82-a14ce4c57494" (UID: "90fdafa5-dc9b-4911-9f82-a14ce4c57494"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:07.165838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.165816    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff" pod "90fdafa5-dc9b-4911-9f82-a14ce4c57494" (UID: "90fdafa5-dc9b-4911-9f82-a14ce4c57494"). InnerVolumeSpecName "kube-api-access-mldff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:07.165939 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.165909    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca" pod "90fdafa5-dc9b-4911-9f82-a14ce4c57494" (UID: "90fdafa5-dc9b-4911-9f82-a14ce4c57494"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:07.264970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.264941    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:07.264970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.264972    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldff\" (UniqueName: \"kubernetes.io/projected/90fdafa5-dc9b-4911-9f82-a14ce4c57494-kube-api-access-mldff\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:07.265172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.264986    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fdafa5-dc9b-4911-9f82-a14ce4c57494-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:07.926733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.926702    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"
Oct 31 00:35:07.926903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.926741    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz" event={"ID":"90fdafa5-dc9b-4911-9f82-a14ce4c57494","Type":"ContainerDied","Data":"23f9ff3797cb579d4bfcbc35c4758223935222b15b5267a42ccf9b0f82a3d2ee"}
Oct 31 00:35:07.926903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:07.926793    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23f9ff3797cb579d4bfcbc35c4758223935222b15b5267a42ccf9b0f82a3d2ee"
Oct 31 00:35:12.025795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:12.025761    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"]
Oct 31 00:35:12.028289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:12.028268    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"]
Oct 31 00:35:12.033282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:12.033246    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5613e-4kmwf"]
Oct 31 00:35:12.035511 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:12.035486    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4377892-66t8n"]
Oct 31 00:35:12.193174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:12.193145    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371d532e-e847-418a-a63e-990af75b81a7" path="/var/lib/kubelet/pods/371d532e-e847-418a-a63e-990af75b81a7/volumes"
Oct 31 00:35:12.193588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:12.193575    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e27c39f-c611-489c-b51a-5fce76e4f52d" path="/var/lib/kubelet/pods/4e27c39f-c611-489c-b51a-5fce76e4f52d/volumes"
Oct 31 00:35:21.898423 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.898386    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"]
Oct 31 00:35:21.898936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.898861    5078 cpu_manager.go:414] "containerMap: removing container" podUID="90fdafa5-dc9b-4911-9f82-a14ce4c57494" containerName="revoke-keys"
Oct 31 00:35:21.898936 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.898872    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fdafa5-dc9b-4911-9f82-a14ce4c57494" containerName="revoke-keys"
Oct 31 00:35:21.899015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.898964    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fdafa5-dc9b-4911-9f82-a14ce4c57494" containerName="revoke-keys"
Oct 31 00:35:21.902146 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.902129    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:21.908552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.908525    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:35:21.908724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.908588    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:35:21.908724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.908708    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:35:21.909138 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.909106    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:35:21.910892 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.910373    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"]
Oct 31 00:35:21.955728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.955704    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:21.955866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.955745    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sts8\" (UniqueName: \"kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:21.955913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:21.955891    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.056724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.056693    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.056862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.056747    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.056862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.056776    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sts8\" (UniqueName: \"kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.057509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.057419    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.059210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.059188    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.070055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.070029    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sts8\" (UniqueName: \"kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8\") pod \"maas-api-revoke-sub-a4aef7d7-vsqfm\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.216412 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.216344    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:22.494310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:22.494284    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"]
Oct 31 00:35:22.495321 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:35:22.495295    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6363d85_b182_4fb3_95ec_b54d9e32dba1.slice/crio-88c8d5886a7282cd8b372a01ff9246943e9809faea4f29e10e348878066d1643 WatchSource:0}: Error finding container 88c8d5886a7282cd8b372a01ff9246943e9809faea4f29e10e348878066d1643: Status 404 returned error can't find the container with id 88c8d5886a7282cd8b372a01ff9246943e9809faea4f29e10e348878066d1643
Oct 31 00:35:23.062390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:23.062306    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm" event={"ID":"b6363d85-b182-4fb3-95ec-b54d9e32dba1","Type":"ContainerStarted","Data":"60c653f5b04bc08c81deaee2e2f65cb5c0c29d3f1b57b93bec2ea233875ae0f7"}
Oct 31 00:35:23.062390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:23.062366    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm" event={"ID":"b6363d85-b182-4fb3-95ec-b54d9e32dba1","Type":"ContainerStarted","Data":"88c8d5886a7282cd8b372a01ff9246943e9809faea4f29e10e348878066d1643"}
Oct 31 00:35:23.086518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:23.086454    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm" podStartSLOduration=2.086441864 podStartE2EDuration="2.086441864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:23.084205683 +0000 UTC m=+2491.030576399" watchObservedRunningTime="2026-10-31 00:35:23.086441864 +0000 UTC m=+2491.032812581"
Oct 31 00:35:24.023668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:24.023636    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"]
Oct 31 00:35:24.026382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:24.026359    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead48221-524q5"]
Oct 31 00:35:24.070392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:24.070360    5078 generic.go:356] "Generic (PLEG): container finished" podID="b6363d85-b182-4fb3-95ec-b54d9e32dba1" containerID="60c653f5b04bc08c81deaee2e2f65cb5c0c29d3f1b57b93bec2ea233875ae0f7" exitCode=0
Oct 31 00:35:24.070793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:24.070396    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm" event={"ID":"b6363d85-b182-4fb3-95ec-b54d9e32dba1","Type":"ContainerDied","Data":"60c653f5b04bc08c81deaee2e2f65cb5c0c29d3f1b57b93bec2ea233875ae0f7"}
Oct 31 00:35:24.193580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:24.193541    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c732715-1515-4487-8c97-fe8f7566e72f" path="/var/lib/kubelet/pods/8c732715-1515-4487-8c97-fe8f7566e72f/volumes"
Oct 31 00:35:25.215636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.215611    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:25.286185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.286159    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8\" (UniqueName: \"kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8\") pod \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") "
Oct 31 00:35:25.286343 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.286194    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca\") pod \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") "
Oct 31 00:35:25.286400 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.286366    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca\") pod \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\" (UID: \"b6363d85-b182-4fb3-95ec-b54d9e32dba1\") "
Oct 31 00:35:25.286640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.286609    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca" pod "b6363d85-b182-4fb3-95ec-b54d9e32dba1" (UID: "b6363d85-b182-4fb3-95ec-b54d9e32dba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:25.286779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.286762    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:25.288496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.288473    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca" pod "b6363d85-b182-4fb3-95ec-b54d9e32dba1" (UID: "b6363d85-b182-4fb3-95ec-b54d9e32dba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:25.288601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.288553    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8" pod "b6363d85-b182-4fb3-95ec-b54d9e32dba1" (UID: "b6363d85-b182-4fb3-95ec-b54d9e32dba1"). InnerVolumeSpecName "kube-api-access-4sts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:25.387635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.387611    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6363d85-b182-4fb3-95ec-b54d9e32dba1-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:25.387635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:25.387632    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sts8\" (UniqueName: \"kubernetes.io/projected/b6363d85-b182-4fb3-95ec-b54d9e32dba1-kube-api-access-4sts8\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:26.083332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:26.083302    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"
Oct 31 00:35:26.083521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:26.083333    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm" event={"ID":"b6363d85-b182-4fb3-95ec-b54d9e32dba1","Type":"ContainerDied","Data":"88c8d5886a7282cd8b372a01ff9246943e9809faea4f29e10e348878066d1643"}
Oct 31 00:35:26.083521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:26.083382    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88c8d5886a7282cd8b372a01ff9246943e9809faea4f29e10e348878066d1643"
Oct 31 00:35:31.025401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:31.025365    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"]
Oct 31 00:35:31.029579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:31.029553    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"]
Oct 31 00:35:31.032284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:31.032123    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedb204d-84xql"]
Oct 31 00:35:31.034775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:31.034756    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e853c3af-xccvc"]
Oct 31 00:35:32.193625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:32.193586    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eea3fe1-8dc8-4c63-baf5-5891e1a0268e" path="/var/lib/kubelet/pods/7eea3fe1-8dc8-4c63-baf5-5891e1a0268e/volumes"
Oct 31 00:35:32.194124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:32.194107    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdf4c03-7887-4170-bf7d-394732ef30dd" path="/var/lib/kubelet/pods/dcdf4c03-7887-4170-bf7d-394732ef30dd/volumes"
Oct 31 00:35:36.465483 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:35:36.465411    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:35:40.968283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.968246    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"]
Oct 31 00:35:40.968721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.968707    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b6363d85-b182-4fb3-95ec-b54d9e32dba1" containerName="revoke-keys"
Oct 31 00:35:40.968770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.968723    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6363d85-b182-4fb3-95ec-b54d9e32dba1" containerName="revoke-keys"
Oct 31 00:35:40.968815 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.968808    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6363d85-b182-4fb3-95ec-b54d9e32dba1" containerName="revoke-keys"
Oct 31 00:35:40.973589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.973567    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:40.979174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.979147    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"]
Oct 31 00:35:40.980441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.980416    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:35:40.980441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.980435    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:35:40.980441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.980425    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:35:40.980657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:40.980493    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:35:41.086758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.086721    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.086758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.086764    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.086975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.086792    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vv9l\" (UniqueName: \"kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.187577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.187539    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.187791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.187594    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.187791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.187648    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vv9l\" (UniqueName: \"kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.188250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.188214    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.190182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.190155    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.201220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.201190    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vv9l\" (UniqueName: \"kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l\") pod \"maas-api-revoke-sub-79805ca9-wn5wb\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.289551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.289449    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:41.567838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:41.567811    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"]
Oct 31 00:35:41.568110 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:35:41.568081    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1c6ffa_125c_4035_86df_a3a79fc2db6a.slice/crio-1949e146225f46470203981cc1062703051897f8dde6aff6b6fde4cda1edd204 WatchSource:0}: Error finding container 1949e146225f46470203981cc1062703051897f8dde6aff6b6fde4cda1edd204: Status 404 returned error can't find the container with id 1949e146225f46470203981cc1062703051897f8dde6aff6b6fde4cda1edd204
Oct 31 00:35:42.195518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:42.195484    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb" event={"ID":"3c1c6ffa-125c-4035-86df-a3a79fc2db6a","Type":"ContainerStarted","Data":"9e96d3e7383568fe0063564f546557d4e5a732432c59ce07274811a701844f6f"}
Oct 31 00:35:42.195928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:42.195541    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb" event={"ID":"3c1c6ffa-125c-4035-86df-a3a79fc2db6a","Type":"ContainerStarted","Data":"1949e146225f46470203981cc1062703051897f8dde6aff6b6fde4cda1edd204"}
Oct 31 00:35:42.217249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:42.217202    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb" podStartSLOduration=2.217189143 podStartE2EDuration="2.217189143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:42.215843778 +0000 UTC m=+2510.162214514" watchObservedRunningTime="2026-10-31 00:35:42.217189143 +0000 UTC m=+2510.163559859"
Oct 31 00:35:43.203086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:43.203056    5078 generic.go:356] "Generic (PLEG): container finished" podID="3c1c6ffa-125c-4035-86df-a3a79fc2db6a" containerID="9e96d3e7383568fe0063564f546557d4e5a732432c59ce07274811a701844f6f" exitCode=0
Oct 31 00:35:43.203525 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:43.203145    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb" event={"ID":"3c1c6ffa-125c-4035-86df-a3a79fc2db6a","Type":"ContainerDied","Data":"9e96d3e7383568fe0063564f546557d4e5a732432c59ce07274811a701844f6f"}
Oct 31 00:35:44.355656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.355628    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:44.420211 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.420178    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca\") pod \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") "
Oct 31 00:35:44.420381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.420243    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca\") pod \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") "
Oct 31 00:35:44.420381 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.420349    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l\" (UniqueName: \"kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l\") pod \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\" (UID: \"3c1c6ffa-125c-4035-86df-a3a79fc2db6a\") "
Oct 31 00:35:44.420635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.420609    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca" pod "3c1c6ffa-125c-4035-86df-a3a79fc2db6a" (UID: "3c1c6ffa-125c-4035-86df-a3a79fc2db6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:44.420758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.420738    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:44.422638 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.422614    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l" pod "3c1c6ffa-125c-4035-86df-a3a79fc2db6a" (UID: "3c1c6ffa-125c-4035-86df-a3a79fc2db6a"). InnerVolumeSpecName "kube-api-access-6vv9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:44.422738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.422677    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca" pod "3c1c6ffa-125c-4035-86df-a3a79fc2db6a" (UID: "3c1c6ffa-125c-4035-86df-a3a79fc2db6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:44.521705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.521624    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vv9l\" (UniqueName: \"kubernetes.io/projected/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-kube-api-access-6vv9l\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:44.521705 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:44.521657    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1c6ffa-125c-4035-86df-a3a79fc2db6a-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:45.216723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:45.216688    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"
Oct 31 00:35:45.216921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:45.216725    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb" event={"ID":"3c1c6ffa-125c-4035-86df-a3a79fc2db6a","Type":"ContainerDied","Data":"1949e146225f46470203981cc1062703051897f8dde6aff6b6fde4cda1edd204"}
Oct 31 00:35:45.216921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:45.216781    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1949e146225f46470203981cc1062703051897f8dde6aff6b6fde4cda1edd204"
Oct 31 00:35:56.356664 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:56.356610    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:35:56.357324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:56.356673    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:35:56.382945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:56.382537    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:35:56.382945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:56.382605    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:35:56.477686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:56.477632    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:35:56.478007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:56.477949    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:35:58.036602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.036569    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"]
Oct 31 00:35:58.049177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.049145    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e6697a-xn5lm"]
Oct 31 00:35:58.204642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.204601    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0972488-0d0d-482e-8bda-9fa193fbf1ef" path="/var/lib/kubelet/pods/f0972488-0d0d-482e-8bda-9fa193fbf1ef/volumes"
Oct 31 00:35:58.893265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.893226    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"]
Oct 31 00:35:58.893696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.893681    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3c1c6ffa-125c-4035-86df-a3a79fc2db6a" containerName="revoke-keys"
Oct 31 00:35:58.893754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.893698    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1c6ffa-125c-4035-86df-a3a79fc2db6a" containerName="revoke-keys"
Oct 31 00:35:58.893802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.893793    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1c6ffa-125c-4035-86df-a3a79fc2db6a" containerName="revoke-keys"
Oct 31 00:35:58.897362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.897342    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:58.904856 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.904830    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:35:58.905007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.904923    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:35:58.905007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.904962    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:35:58.905007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.905008    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:35:58.906688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.906661    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"]
Oct 31 00:35:58.989229 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.989189    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:58.989411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.989237    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:58.989411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:58.989270    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjj9\" (UniqueName: \"kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.090558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.090522    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.090973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.090575    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjj9\" (UniqueName: \"kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.090973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.090669    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.091279 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.091262    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.093065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.093043    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.104158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.104129    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjj9\" (UniqueName: \"kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9\") pod \"maas-api-revoke-sub-0b7e8edb-j9cbc\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.212518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.212405    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:35:59.489427 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:35:59.489403    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"]
Oct 31 00:35:59.491841 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:35:59.491815    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea95e540_d65d_454d_8ba3_85fd94573b11.slice/crio-1b61cd0fa492164274d02c46cf1afb049914d50e3b6564e9405c8d2c9b03b59d WatchSource:0}: Error finding container 1b61cd0fa492164274d02c46cf1afb049914d50e3b6564e9405c8d2c9b03b59d: Status 404 returned error can't find the container with id 1b61cd0fa492164274d02c46cf1afb049914d50e3b6564e9405c8d2c9b03b59d
Oct 31 00:36:00.350015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:00.349981    5078 generic.go:356] "Generic (PLEG): container finished" podID="ea95e540-d65d-454d-8ba3-85fd94573b11" containerID="133dffdba9797ed022acae74c2d6fb3c704f090d63e6ed4407dc08cfc059cb95" exitCode=0
Oct 31 00:36:00.350416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:00.350043    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc" event={"ID":"ea95e540-d65d-454d-8ba3-85fd94573b11","Type":"ContainerDied","Data":"133dffdba9797ed022acae74c2d6fb3c704f090d63e6ed4407dc08cfc059cb95"}
Oct 31 00:36:00.350416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:00.350100    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc" event={"ID":"ea95e540-d65d-454d-8ba3-85fd94573b11","Type":"ContainerStarted","Data":"1b61cd0fa492164274d02c46cf1afb049914d50e3b6564e9405c8d2c9b03b59d"}
Oct 31 00:36:01.495346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.495317    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:36:01.513509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.513487    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9\" (UniqueName: \"kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9\") pod \"ea95e540-d65d-454d-8ba3-85fd94573b11\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") "
Oct 31 00:36:01.513636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.513518    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca\") pod \"ea95e540-d65d-454d-8ba3-85fd94573b11\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") "
Oct 31 00:36:01.513636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.513593    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca\") pod \"ea95e540-d65d-454d-8ba3-85fd94573b11\" (UID: \"ea95e540-d65d-454d-8ba3-85fd94573b11\") "
Oct 31 00:36:01.513984 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.513961    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca" pod "ea95e540-d65d-454d-8ba3-85fd94573b11" (UID: "ea95e540-d65d-454d-8ba3-85fd94573b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:01.514101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.514073    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:01.515991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.515971    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9" pod "ea95e540-d65d-454d-8ba3-85fd94573b11" (UID: "ea95e540-d65d-454d-8ba3-85fd94573b11"). InnerVolumeSpecName "kube-api-access-hsjj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:01.516059 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.516034    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca" pod "ea95e540-d65d-454d-8ba3-85fd94573b11" (UID: "ea95e540-d65d-454d-8ba3-85fd94573b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:01.614917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.614843    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsjj9\" (UniqueName: \"kubernetes.io/projected/ea95e540-d65d-454d-8ba3-85fd94573b11-kube-api-access-hsjj9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:01.614917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:01.614870    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea95e540-d65d-454d-8ba3-85fd94573b11-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:02.365310 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:02.365283    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"
Oct 31 00:36:02.365529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:02.365277    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc" event={"ID":"ea95e540-d65d-454d-8ba3-85fd94573b11","Type":"ContainerDied","Data":"1b61cd0fa492164274d02c46cf1afb049914d50e3b6564e9405c8d2c9b03b59d"}
Oct 31 00:36:02.365529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:02.365409    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b61cd0fa492164274d02c46cf1afb049914d50e3b6564e9405c8d2c9b03b59d"
Oct 31 00:36:06.386635 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:36:06.386600    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:36:09.630217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.630182    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"]
Oct 31 00:36:09.630842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.630821    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ea95e540-d65d-454d-8ba3-85fd94573b11" containerName="revoke-keys"
Oct 31 00:36:09.630894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.630843    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea95e540-d65d-454d-8ba3-85fd94573b11" containerName="revoke-keys"
Oct 31 00:36:09.630973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.630935    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea95e540-d65d-454d-8ba3-85fd94573b11" containerName="revoke-keys"
Oct 31 00:36:09.634455 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.634436    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.641022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.640993    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:36:09.645719 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.641878    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:36:09.650390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.650361    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:36:09.650589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.650572    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:36:09.656707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.656677    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"]
Oct 31 00:36:09.695361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.695298    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.695603 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.695488    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxtj\" (UniqueName: \"kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.695722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.695644    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.796790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.796745    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.796997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.796816    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.796997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.796888    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxtj\" (UniqueName: \"kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.797731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.797698    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.799341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.799321    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.813886 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.813863    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxtj\" (UniqueName: \"kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj\") pod \"maas-api-revoke-sub-3d4156f1-jptbq\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:09.966185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:09.966155    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:10.249201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:10.249177    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"]
Oct 31 00:36:10.250693 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:36:10.250667    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5e8e69_0b79_4720_af19_651b6d61b47e.slice/crio-aaa268f5e66de595a19e56ed511cf2d526a1de3eb04e2e1695b990dd9a32fe44 WatchSource:0}: Error finding container aaa268f5e66de595a19e56ed511cf2d526a1de3eb04e2e1695b990dd9a32fe44: Status 404 returned error can't find the container with id aaa268f5e66de595a19e56ed511cf2d526a1de3eb04e2e1695b990dd9a32fe44
Oct 31 00:36:10.417144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:10.417093    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq" event={"ID":"0a5e8e69-0b79-4720-af19-651b6d61b47e","Type":"ContainerStarted","Data":"aaa268f5e66de595a19e56ed511cf2d526a1de3eb04e2e1695b990dd9a32fe44"}
Oct 31 00:36:11.424543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:11.424425    5078 generic.go:356] "Generic (PLEG): container finished" podID="0a5e8e69-0b79-4720-af19-651b6d61b47e" containerID="c08ef62516d9dffb36ba81aed1df969a8ec0a7484a1c331d449f3137f620285a" exitCode=0
Oct 31 00:36:11.424543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:11.424478    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq" event={"ID":"0a5e8e69-0b79-4720-af19-651b6d61b47e","Type":"ContainerDied","Data":"c08ef62516d9dffb36ba81aed1df969a8ec0a7484a1c331d449f3137f620285a"}
Oct 31 00:36:12.586053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.586029    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:12.627548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.627512    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca\") pod \"0a5e8e69-0b79-4720-af19-651b6d61b47e\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") "
Oct 31 00:36:12.627733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.627618    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj\" (UniqueName: \"kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj\") pod \"0a5e8e69-0b79-4720-af19-651b6d61b47e\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") "
Oct 31 00:36:12.627733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.627681    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca\") pod \"0a5e8e69-0b79-4720-af19-651b6d61b47e\" (UID: \"0a5e8e69-0b79-4720-af19-651b6d61b47e\") "
Oct 31 00:36:12.627970 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.627943    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca" pod "0a5e8e69-0b79-4720-af19-651b6d61b47e" (UID: "0a5e8e69-0b79-4720-af19-651b6d61b47e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:12.628101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.628078    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.629957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.629936    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca" pod "0a5e8e69-0b79-4720-af19-651b6d61b47e" (UID: "0a5e8e69-0b79-4720-af19-651b6d61b47e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:12.630342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.630328    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj" pod "0a5e8e69-0b79-4720-af19-651b6d61b47e" (UID: "0a5e8e69-0b79-4720-af19-651b6d61b47e"). InnerVolumeSpecName "kube-api-access-pnxtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:12.729450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.729366    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxtj\" (UniqueName: \"kubernetes.io/projected/0a5e8e69-0b79-4720-af19-651b6d61b47e-kube-api-access-pnxtj\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.729450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:12.729395    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5e8e69-0b79-4720-af19-651b6d61b47e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:13.437793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:13.437759    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq" event={"ID":"0a5e8e69-0b79-4720-af19-651b6d61b47e","Type":"ContainerDied","Data":"aaa268f5e66de595a19e56ed511cf2d526a1de3eb04e2e1695b990dd9a32fe44"}
Oct 31 00:36:13.437793 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:13.437786    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"
Oct 31 00:36:13.438028 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:13.437806    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaa268f5e66de595a19e56ed511cf2d526a1de3eb04e2e1695b990dd9a32fe44"
Oct 31 00:36:30.025317 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:30.025280    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"]
Oct 31 00:36:30.028629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:30.028609    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb1795f-6rv9n"]
Oct 31 00:36:30.193300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:30.193265    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6be281-7a75-4f13-948a-204d33efcb7f" path="/var/lib/kubelet/pods/ef6be281-7a75-4f13-948a-204d33efcb7f/volumes"
Oct 31 00:36:31.052897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.052866    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"]
Oct 31 00:36:31.055537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.055510    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jptbq"]
Oct 31 00:36:31.093906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.093881    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"]
Oct 31 00:36:31.094368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.094350    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0a5e8e69-0b79-4720-af19-651b6d61b47e" containerName="revoke-keys"
Oct 31 00:36:31.094368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.094367    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5e8e69-0b79-4720-af19-651b6d61b47e" containerName="revoke-keys"
Oct 31 00:36:31.094556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.094443    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5e8e69-0b79-4720-af19-651b6d61b47e" containerName="revoke-keys"
Oct 31 00:36:31.097733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.097719    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.106166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.106144    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:36:31.106331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.106220    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:36:31.109897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.109872    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:36:31.110024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.109961    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:36:31.111672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.111654    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"]
Oct 31 00:36:31.189392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.189355    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.189579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.189427    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpgn\" (UniqueName: \"kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.189579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.189527    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.290874 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.290840    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.291055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.290904    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpgn\" (UniqueName: \"kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.291055 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.290939    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.291556 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.291537    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.293546 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.293519    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.304072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.304022    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpgn\" (UniqueName: \"kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn\") pod \"maas-api-revoke-sub-3d4156f1-gnxxd\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.413008 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.412947    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:31.696212 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.696180    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"]
Oct 31 00:36:31.697518 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:36:31.697490    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17157e79_c8cd_496a_b200_29b8d5827463.slice/crio-c86e17a160e6df5f9cb7dec9fbf74430ee2f689c8a837132d71c15730dd8364a WatchSource:0}: Error finding container c86e17a160e6df5f9cb7dec9fbf74430ee2f689c8a837132d71c15730dd8364a: Status 404 returned error can't find the container with id c86e17a160e6df5f9cb7dec9fbf74430ee2f689c8a837132d71c15730dd8364a
Oct 31 00:36:31.767551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.767528    5078 scope.go:122] "RemoveContainer" containerID="084678dc3da9fb328054e4646a29117ac83cafb3812578714d0dd6dce0993d56"
Oct 31 00:36:31.776920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.776898    5078 scope.go:122] "RemoveContainer" containerID="bcd65622019f4ba5a2faffee85aa803f33adc8101476bcdbaa9a335a147846c8"
Oct 31 00:36:31.785043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.785020    5078 scope.go:122] "RemoveContainer" containerID="31df9b0a2cf1fbd49840ae0965e048620809e509b1580608cfbaa372852e01be"
Oct 31 00:36:31.793128 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.793105    5078 scope.go:122] "RemoveContainer" containerID="a8424a436e1a69ee2bb7bdd461d6a6f863a3b054b5c903a7d39fcc66fea3c965"
Oct 31 00:36:31.802069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.802049    5078 scope.go:122] "RemoveContainer" containerID="86bb60ab180293e9d2dd94855ef954e1f09ab8fa80b45e5eeaace039a57bc486"
Oct 31 00:36:31.810044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.810024    5078 scope.go:122] "RemoveContainer" containerID="8ae0b072798b4abb138d89854dc3efad10f5ff9d5fa632c7108a5056aba84121"
Oct 31 00:36:31.818014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.817969    5078 scope.go:122] "RemoveContainer" containerID="9d3d193131631eb40006edcd3f116abfcaaf9ab6d9825cece92886b82d6faaba"
Oct 31 00:36:31.825968 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:31.825952    5078 scope.go:122] "RemoveContainer" containerID="4a5bd963d2e81e0f50f126a5ccab0db1a255da61e03dcc807371114c974b1175"
Oct 31 00:36:32.195067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:32.195035    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5e8e69-0b79-4720-af19-651b6d61b47e" path="/var/lib/kubelet/pods/0a5e8e69-0b79-4720-af19-651b6d61b47e/volumes"
Oct 31 00:36:32.624812 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:32.624786    5078 generic.go:356] "Generic (PLEG): container finished" podID="17157e79-c8cd-496a-b200-29b8d5827463" containerID="6ab8e306f9b6b1ed8c6dec713e1d2c274dd7842ee5561085c0e8b220588ae8a9" exitCode=0
Oct 31 00:36:32.624946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:32.624865    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd" event={"ID":"17157e79-c8cd-496a-b200-29b8d5827463","Type":"ContainerDied","Data":"6ab8e306f9b6b1ed8c6dec713e1d2c274dd7842ee5561085c0e8b220588ae8a9"}
Oct 31 00:36:32.624946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:32.624915    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd" event={"ID":"17157e79-c8cd-496a-b200-29b8d5827463","Type":"ContainerStarted","Data":"c86e17a160e6df5f9cb7dec9fbf74430ee2f689c8a837132d71c15730dd8364a"}
Oct 31 00:36:33.778935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.778910    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:33.813865 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.813841    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca\") pod \"17157e79-c8cd-496a-b200-29b8d5827463\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") "
Oct 31 00:36:33.814045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.813934    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca\") pod \"17157e79-c8cd-496a-b200-29b8d5827463\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") "
Oct 31 00:36:33.814045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.813973    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn\" (UniqueName: \"kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn\") pod \"17157e79-c8cd-496a-b200-29b8d5827463\" (UID: \"17157e79-c8cd-496a-b200-29b8d5827463\") "
Oct 31 00:36:33.814250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.814224    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca" pod "17157e79-c8cd-496a-b200-29b8d5827463" (UID: "17157e79-c8cd-496a-b200-29b8d5827463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:33.814333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.814316    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:33.819979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.819940    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn" pod "17157e79-c8cd-496a-b200-29b8d5827463" (UID: "17157e79-c8cd-496a-b200-29b8d5827463"). InnerVolumeSpecName "kube-api-access-jfpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:33.821758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.821725    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca" pod "17157e79-c8cd-496a-b200-29b8d5827463" (UID: "17157e79-c8cd-496a-b200-29b8d5827463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:33.915359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.915331    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17157e79-c8cd-496a-b200-29b8d5827463-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:33.915359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:33.915359    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfpgn\" (UniqueName: \"kubernetes.io/projected/17157e79-c8cd-496a-b200-29b8d5827463-kube-api-access-jfpgn\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:36:34.637408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:34.637349    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"
Oct 31 00:36:34.637408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:34.637384    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd" event={"ID":"17157e79-c8cd-496a-b200-29b8d5827463","Type":"ContainerDied","Data":"c86e17a160e6df5f9cb7dec9fbf74430ee2f689c8a837132d71c15730dd8364a"}
Oct 31 00:36:34.637629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:34.637428    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86e17a160e6df5f9cb7dec9fbf74430ee2f689c8a837132d71c15730dd8364a"
Oct 31 00:36:36.386251 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:36:36.386213    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:36:53.026159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:53.026119    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"]
Oct 31 00:36:53.029306 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:53.029285    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a0e3f3-fsg8w"]
Oct 31 00:36:54.194504 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:36:54.194442    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3026d1d2-6db0-4b98-a40d-f0486afa9ef9" path="/var/lib/kubelet/pods/3026d1d2-6db0-4b98-a40d-f0486afa9ef9/volumes"
Oct 31 00:37:03.367906 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.367876    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"]
Oct 31 00:37:03.368390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.368320    5078 cpu_manager.go:414] "containerMap: removing container" podUID="17157e79-c8cd-496a-b200-29b8d5827463" containerName="revoke-keys"
Oct 31 00:37:03.368390 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.368331    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17157e79-c8cd-496a-b200-29b8d5827463" containerName="revoke-keys"
Oct 31 00:37:03.368493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.368397    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="17157e79-c8cd-496a-b200-29b8d5827463" containerName="revoke-keys"
Oct 31 00:37:03.371896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.371873    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.378252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.378213    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:37:03.380594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.380566    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:37:03.380725 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.380692    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:37:03.380818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.380802    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:37:03.383841 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.383823    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"]
Oct 31 00:37:03.513608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.513573    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.513764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.513615    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.513764 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.513752    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkls\" (UniqueName: \"kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.614471 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.614431    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.614626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.614479    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.614626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.614513    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkls\" (UniqueName: \"kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.615051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.615029    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.616963 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.616933    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.627905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.627857    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkls\" (UniqueName: \"kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls\") pod \"maas-api-revoke-sub-e2fc9c5b-dvqm5\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.687185 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.687161    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:03.964167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:03.964143    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"]
Oct 31 00:37:03.965075 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:37:03.965047    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81e3386_a20e_4eb8_89c8_7dea59d56c97.slice/crio-06e1272cbce0e457f1e0d84091e497e997a160319afecf7cfea46009b7e1b564 WatchSource:0}: Error finding container 06e1272cbce0e457f1e0d84091e497e997a160319afecf7cfea46009b7e1b564: Status 404 returned error can't find the container with id 06e1272cbce0e457f1e0d84091e497e997a160319afecf7cfea46009b7e1b564
Oct 31 00:37:04.861414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:04.861379    5078 generic.go:356] "Generic (PLEG): container finished" podID="c81e3386-a20e-4eb8-89c8-7dea59d56c97" containerID="ec9a38875d59e1a42fb2b83973f1ec86d1912a759ed924ea19e04ea8d9ce2e60" exitCode=0
Oct 31 00:37:04.861869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:04.861427    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5" event={"ID":"c81e3386-a20e-4eb8-89c8-7dea59d56c97","Type":"ContainerDied","Data":"ec9a38875d59e1a42fb2b83973f1ec86d1912a759ed924ea19e04ea8d9ce2e60"}
Oct 31 00:37:04.861869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:04.861523    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5" event={"ID":"c81e3386-a20e-4eb8-89c8-7dea59d56c97","Type":"ContainerStarted","Data":"06e1272cbce0e457f1e0d84091e497e997a160319afecf7cfea46009b7e1b564"}
Oct 31 00:37:06.020187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.020163    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:06.140921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.140891    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls\" (UniqueName: \"kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls\") pod \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") "
Oct 31 00:37:06.141113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.140934    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca\") pod \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") "
Oct 31 00:37:06.141113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.140967    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca\") pod \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\" (UID: \"c81e3386-a20e-4eb8-89c8-7dea59d56c97\") "
Oct 31 00:37:06.141347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.141324    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca" pod "c81e3386-a20e-4eb8-89c8-7dea59d56c97" (UID: "c81e3386-a20e-4eb8-89c8-7dea59d56c97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:37:06.143217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.143195    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca" pod "c81e3386-a20e-4eb8-89c8-7dea59d56c97" (UID: "c81e3386-a20e-4eb8-89c8-7dea59d56c97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:37:06.143296 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.143275    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls" pod "c81e3386-a20e-4eb8-89c8-7dea59d56c97" (UID: "c81e3386-a20e-4eb8-89c8-7dea59d56c97"). InnerVolumeSpecName "kube-api-access-gkkls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:06.242534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.242510    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkkls\" (UniqueName: \"kubernetes.io/projected/c81e3386-a20e-4eb8-89c8-7dea59d56c97-kube-api-access-gkkls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:06.242534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.242533    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:06.242695 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.242544    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e3386-a20e-4eb8-89c8-7dea59d56c97-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:06.385956 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:37:06.385920    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:37:06.875656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.875622    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5" event={"ID":"c81e3386-a20e-4eb8-89c8-7dea59d56c97","Type":"ContainerDied","Data":"06e1272cbce0e457f1e0d84091e497e997a160319afecf7cfea46009b7e1b564"}
Oct 31 00:37:06.875846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.875674    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e1272cbce0e457f1e0d84091e497e997a160319afecf7cfea46009b7e1b564"
Oct 31 00:37:06.875846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:06.875637    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"
Oct 31 00:37:13.608118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.608084    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"]
Oct 31 00:37:13.608913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.608892    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c81e3386-a20e-4eb8-89c8-7dea59d56c97" containerName="revoke-keys"
Oct 31 00:37:13.608964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.608917    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81e3386-a20e-4eb8-89c8-7dea59d56c97" containerName="revoke-keys"
Oct 31 00:37:13.609256 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.609031    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81e3386-a20e-4eb8-89c8-7dea59d56c97" containerName="revoke-keys"
Oct 31 00:37:13.612701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.612685    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.619489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.618407    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:37:13.619489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.618597    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:37:13.619667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.619607    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:37:13.619722 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.619676    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:37:13.620484 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.620442    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"]
Oct 31 00:37:13.703358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.703328    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.703569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.703388    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdw5\" (UniqueName: \"kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.703569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.703538    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.804105 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.804072    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.804292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.804120    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.804292 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.804176    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdw5\" (UniqueName: \"kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.804883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.804858    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.806564 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.806543    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.817680 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.817653    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdw5\" (UniqueName: \"kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5\") pod \"maas-api-revoke-sub-5a29efb3-c4tmb\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:13.927621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:13.927590    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:14.212177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:14.212138    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"]
Oct 31 00:37:14.212355 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:37:14.212333    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3746b532_552e_4399_9802_56a77b51a29c.slice/crio-d047322abd57ace4f3f0b4df9bd511c3103b3f931df3f3d20774bc34a3fed9e0 WatchSource:0}: Error finding container d047322abd57ace4f3f0b4df9bd511c3103b3f931df3f3d20774bc34a3fed9e0: Status 404 returned error can't find the container with id d047322abd57ace4f3f0b4df9bd511c3103b3f931df3f3d20774bc34a3fed9e0
Oct 31 00:37:14.928338 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:14.928294    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb" event={"ID":"3746b532-552e-4399-9802-56a77b51a29c","Type":"ContainerStarted","Data":"13c9e0fc979591e4f7a3ee3c27de0d9042ea07cb51e9c865973f5642938cff44"}
Oct 31 00:37:14.928891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:14.928371    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb" event={"ID":"3746b532-552e-4399-9802-56a77b51a29c","Type":"ContainerStarted","Data":"d047322abd57ace4f3f0b4df9bd511c3103b3f931df3f3d20774bc34a3fed9e0"}
Oct 31 00:37:14.952831 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:14.952779    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb" podStartSLOduration=1.952761432 podStartE2EDuration="1.952761432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:14.951769345 +0000 UTC m=+2602.898140062" watchObservedRunningTime="2026-10-31 00:37:14.952761432 +0000 UTC m=+2602.899132150"
Oct 31 00:37:15.955495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:15.955410    5078 generic.go:356] "Generic (PLEG): container finished" podID="3746b532-552e-4399-9802-56a77b51a29c" containerID="13c9e0fc979591e4f7a3ee3c27de0d9042ea07cb51e9c865973f5642938cff44" exitCode=0
Oct 31 00:37:15.956248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:15.956221    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb" event={"ID":"3746b532-552e-4399-9802-56a77b51a29c","Type":"ContainerDied","Data":"13c9e0fc979591e4f7a3ee3c27de0d9042ea07cb51e9c865973f5642938cff44"}
Oct 31 00:37:17.316828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.316803    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:17.463290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.463260    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca\") pod \"3746b532-552e-4399-9802-56a77b51a29c\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") "
Oct 31 00:37:17.463495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.463319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca\") pod \"3746b532-552e-4399-9802-56a77b51a29c\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") "
Oct 31 00:37:17.463495 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.463403    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5\" (UniqueName: \"kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5\") pod \"3746b532-552e-4399-9802-56a77b51a29c\" (UID: \"3746b532-552e-4399-9802-56a77b51a29c\") "
Oct 31 00:37:17.463734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.463709    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca" pod "3746b532-552e-4399-9802-56a77b51a29c" (UID: "3746b532-552e-4399-9802-56a77b51a29c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:37:17.465773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.465743    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5" pod "3746b532-552e-4399-9802-56a77b51a29c" (UID: "3746b532-552e-4399-9802-56a77b51a29c"). InnerVolumeSpecName "kube-api-access-5cdw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:17.465773 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.465754    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca" pod "3746b532-552e-4399-9802-56a77b51a29c" (UID: "3746b532-552e-4399-9802-56a77b51a29c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:37:17.564768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.564667    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cdw5\" (UniqueName: \"kubernetes.io/projected/3746b532-552e-4399-9802-56a77b51a29c-kube-api-access-5cdw5\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:17.564768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.564710    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3746b532-552e-4399-9802-56a77b51a29c-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:17.564768 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.564725    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3746b532-552e-4399-9802-56a77b51a29c-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:17.971025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.970991    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"
Oct 31 00:37:17.971025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.971017    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb" event={"ID":"3746b532-552e-4399-9802-56a77b51a29c","Type":"ContainerDied","Data":"d047322abd57ace4f3f0b4df9bd511c3103b3f931df3f3d20774bc34a3fed9e0"}
Oct 31 00:37:17.971253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:17.971070    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d047322abd57ace4f3f0b4df9bd511c3103b3f931df3f3d20774bc34a3fed9e0"
Oct 31 00:37:31.881186 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:31.881147    5078 scope.go:122] "RemoveContainer" containerID="6282c41fcdc477da840df51d309ab2b1dbf11181ad200a2b6ee4393df3e1a303"
Oct 31 00:37:31.892401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:31.892378    5078 scope.go:122] "RemoveContainer" containerID="15af02a777eee73c3f7a0c740b3be12358ba6212e2f9a4b5f406555ddfaf1676"
Oct 31 00:37:31.902031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:31.902011    5078 scope.go:122] "RemoveContainer" containerID="f16c2d69d2ab8a3d2998dc773b904651e1b3ecb80558d0d7a49bcf74f1a107f7"
Oct 31 00:37:31.910765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:31.910748    5078 scope.go:122] "RemoveContainer" containerID="06d0c0b2e448153d426e905937e67ea8b4774e3c0ec6999cb345dc7d35e28ddc"
Oct 31 00:37:31.918679 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:31.918663    5078 scope.go:122] "RemoveContainer" containerID="0615da6a5189950fcbcdbef0db8d5e72a1f539e6c352335a9f204e306be13b27"
Oct 31 00:37:31.926386 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:31.926362    5078 scope.go:122] "RemoveContainer" containerID="23488c43f37ac5ab81a1f1a312e38a2c54a22751fc3e67d974a2746e179188b8"
Oct 31 00:37:36.389903 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:37:36.389864    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:37:43.025403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:43.025372    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"]
Oct 31 00:37:43.028137 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:43.028099    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h9wdq"]
Oct 31 00:37:44.193585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:37:44.193553    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a39bad-e1f2-45a4-a710-6e346c67f942" path="/var/lib/kubelet/pods/f4a39bad-e1f2-45a4-a710-6e346c67f942/volumes"
Oct 31 00:38:06.386054 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:38:06.386007    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:38:31.968064 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:31.968034    5078 scope.go:122] "RemoveContainer" containerID="931652e4201625e5f63db19f4e12cf1f8abeab02fd97c7af62144cb03826a2e4"
Oct 31 00:38:31.977347 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:31.977326    5078 scope.go:122] "RemoveContainer" containerID="ea7eb9d89216e3b80345c378d159f0f4b5595bb3e31b0f544c7d0d784d6c7df6"
Oct 31 00:38:36.386066 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:38:36.386027    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:38:40.035492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:40.035428    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"]
Oct 31 00:38:40.037703 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:40.037681    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2qr8f"]
Oct 31 00:38:40.193208 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:40.193175    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bed518f-78fb-48c9-ab10-2577679aa963" path="/var/lib/kubelet/pods/4bed518f-78fb-48c9-ab10-2577679aa963/volumes"
Oct 31 00:38:54.534770 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:54.534738    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:38:54.536738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:54.536719    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:38:54.540275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:54.540244    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:38:54.542433 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:54.542410    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:38:59.024857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:59.024819    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"]
Oct 31 00:38:59.031190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:38:59.031166    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-px52p"]
Oct 31 00:39:00.193806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:00.193772    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efcef99b-70f8-47ee-94cb-aeeef9cf339f" path="/var/lib/kubelet/pods/efcef99b-70f8-47ee-94cb-aeeef9cf339f/volumes"
Oct 31 00:39:06.385778 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:39:06.385734    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:39:10.035022 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:10.034983    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"]
Oct 31 00:39:10.037415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:10.037391    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d6wpz"]
Oct 31 00:39:10.194380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:10.194347    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa8dce9-848b-437b-8d9e-e146549d6ef6" path="/var/lib/kubelet/pods/0aa8dce9-848b-437b-8d9e-e146549d6ef6/volumes"
Oct 31 00:39:32.002670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:32.002593    5078 scope.go:122] "RemoveContainer" containerID="3d414de2984395d6cee9151ae8aed3bc78f2c5a1000115c62afa8cacbbc04e6f"
Oct 31 00:39:36.028377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:36.028342    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"]
Oct 31 00:39:36.031373 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:36.031355    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7lxj"]
Oct 31 00:39:36.193216 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:36.193181    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d2694c-c73e-45cb-86c8-71cb366ea06f" path="/var/lib/kubelet/pods/74d2694c-c73e-45cb-86c8-71cb366ea06f/volumes"
Oct 31 00:39:36.386242 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:39:36.386206    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:39:46.025582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:46.025544    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"]
Oct 31 00:39:46.028869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:46.028844    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-m9drm"]
Oct 31 00:39:46.193151 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:39:46.193117    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5" path="/var/lib/kubelet/pods/4ba0d5d6-c8e0-4fed-a4fb-144ece03c5d5/volumes"
Oct 31 00:40:06.386535 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:40:06.386493    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:40:07.028736 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:07.028692    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"]
Oct 31 00:40:07.032316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:07.032293    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gxpgz"]
Oct 31 00:40:08.193836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:08.193800    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fdafa5-dc9b-4911-9f82-a14ce4c57494" path="/var/lib/kubelet/pods/90fdafa5-dc9b-4911-9f82-a14ce4c57494/volumes"
Oct 31 00:40:26.023348 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:26.023307    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"]
Oct 31 00:40:26.027500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:26.027451    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vsqfm"]
Oct 31 00:40:26.193701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:26.193667    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6363d85-b182-4fb3-95ec-b54d9e32dba1" path="/var/lib/kubelet/pods/b6363d85-b182-4fb3-95ec-b54d9e32dba1/volumes"
Oct 31 00:40:32.026876 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:32.026841    5078 scope.go:122] "RemoveContainer" containerID="22ff7b0dfbb5b6890692ef38562807d1b0918122fa339f6510f06f4a4005a17e"
Oct 31 00:40:32.036179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:32.036156    5078 scope.go:122] "RemoveContainer" containerID="0d49ea8d52912847a656a0571432d3098800aa17b56fd9f63df1e21dea8713bd"
Oct 31 00:40:32.044981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:32.044964    5078 scope.go:122] "RemoveContainer" containerID="17eb6ae3e71ef723c4df7fe7d4fd62ab9209afd33baaf9db8384d3b9d23f69dd"
Oct 31 00:40:36.385616 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:40:36.385582    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:40:45.024010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:45.023976    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"]
Oct 31 00:40:45.027071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:45.027048    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wn5wb"]
Oct 31 00:40:46.193362 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:40:46.193329    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1c6ffa-125c-4035-86df-a3a79fc2db6a" path="/var/lib/kubelet/pods/3c1c6ffa-125c-4035-86df-a3a79fc2db6a/volumes"
Oct 31 00:41:02.040941 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:02.040856    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"]
Oct 31 00:41:02.045867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:02.045844    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j9cbc"]
Oct 31 00:41:02.194609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:02.194574    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea95e540-d65d-454d-8ba3-85fd94573b11" path="/var/lib/kubelet/pods/ea95e540-d65d-454d-8ba3-85fd94573b11/volumes"
Oct 31 00:41:06.385668 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:41:06.385633    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:41:32.075048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:32.075017    5078 scope.go:122] "RemoveContainer" containerID="60c653f5b04bc08c81deaee2e2f65cb5c0c29d3f1b57b93bec2ea233875ae0f7"
Oct 31 00:41:32.084282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:32.084265    5078 scope.go:122] "RemoveContainer" containerID="aef0ada785924970d16b7eadb19d04ad0d15b819117ccefecb3e784092f178d8"
Oct 31 00:41:32.092451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:32.092432    5078 scope.go:122] "RemoveContainer" containerID="e9f4e882764cfc7655d90e830b60befebad6338d072157b8f435e508868e7675"
Oct 31 00:41:32.100195 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:32.100174    5078 scope.go:122] "RemoveContainer" containerID="c675eca1fef963aa3cf75f23271a12ec315ab3011727bce22fdff1428fda52e7"
Oct 31 00:41:34.027403 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:34.027359    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"]
Oct 31 00:41:34.029654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:34.029626    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gnxxd"]
Oct 31 00:41:34.192828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:41:34.192797    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17157e79-c8cd-496a-b200-29b8d5827463" path="/var/lib/kubelet/pods/17157e79-c8cd-496a-b200-29b8d5827463/volumes"
Oct 31 00:41:36.386122 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:41:36.386089    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:42:06.385742 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:42:06.385705    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:42:07.025193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:07.025151    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"]
Oct 31 00:42:07.027258 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:07.027238    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dvqm5"]
Oct 31 00:42:08.193275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:08.193237    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81e3386-a20e-4eb8-89c8-7dea59d56c97" path="/var/lib/kubelet/pods/c81e3386-a20e-4eb8-89c8-7dea59d56c97/volumes"
Oct 31 00:42:17.023861 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:17.023827    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"]
Oct 31 00:42:17.027672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:17.027648    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c4tmb"]
Oct 31 00:42:18.192925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:18.192889    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3746b532-552e-4399-9802-56a77b51a29c" path="/var/lib/kubelet/pods/3746b532-552e-4399-9802-56a77b51a29c/volumes"
Oct 31 00:42:30.738850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.738814    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"]
Oct 31 00:42:30.739331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.739269    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3746b532-552e-4399-9802-56a77b51a29c" containerName="revoke-keys"
Oct 31 00:42:30.739331 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.739281    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3746b532-552e-4399-9802-56a77b51a29c" containerName="revoke-keys"
Oct 31 00:42:30.739416 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.739350    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3746b532-552e-4399-9802-56a77b51a29c" containerName="revoke-keys"
Oct 31 00:42:30.742580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.742555    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.750781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.750758    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:42:30.750859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.750840    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:42:30.750914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.750880    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:42:30.750967 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.750951    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:42:30.772800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.772772    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"]
Oct 31 00:42:30.841690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.841659    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.841842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.841699    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp66x\" (UniqueName: \"kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.841842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.841800    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.943223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.943186    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.943382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.943231    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp66x\" (UniqueName: \"kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.943382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.943266    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.943946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.943925    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.945784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.945767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:30.956627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:30.956602    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp66x\" (UniqueName: \"kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x\") pod \"maas-api-revoke-sub-3d4156f1-dsp2q\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:31.056678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:31.056616    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:31.339358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:31.339337    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"]
Oct 31 00:42:31.340715 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:42:31.340688    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86b5a81_4a0c_497c_9890_7ae79322144e.slice/crio-239a3a3b27738b47cbc945486f5206f58255bed857e3e6e9cfaad29d646a31c4 WatchSource:0}: Error finding container 239a3a3b27738b47cbc945486f5206f58255bed857e3e6e9cfaad29d646a31c4: Status 404 returned error can't find the container with id 239a3a3b27738b47cbc945486f5206f58255bed857e3e6e9cfaad29d646a31c4
Oct 31 00:42:31.680037 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:31.680008    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q" event={"ID":"e86b5a81-4a0c-497c-9890-7ae79322144e","Type":"ContainerStarted","Data":"239a3a3b27738b47cbc945486f5206f58255bed857e3e6e9cfaad29d646a31c4"}
Oct 31 00:42:32.132842 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:32.132762    5078 scope.go:122] "RemoveContainer" containerID="c08ef62516d9dffb36ba81aed1df969a8ec0a7484a1c331d449f3137f620285a"
Oct 31 00:42:32.142038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:32.142017    5078 scope.go:122] "RemoveContainer" containerID="9e96d3e7383568fe0063564f546557d4e5a732432c59ce07274811a701844f6f"
Oct 31 00:42:32.150224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:32.150206    5078 scope.go:122] "RemoveContainer" containerID="6ab8e306f9b6b1ed8c6dec713e1d2c274dd7842ee5561085c0e8b220588ae8a9"
Oct 31 00:42:32.158056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:32.158039    5078 scope.go:122] "RemoveContainer" containerID="133dffdba9797ed022acae74c2d6fb3c704f090d63e6ed4407dc08cfc059cb95"
Oct 31 00:42:32.687222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:32.687184    5078 generic.go:356] "Generic (PLEG): container finished" podID="e86b5a81-4a0c-497c-9890-7ae79322144e" containerID="8cf285a8e0b69b196a1daa56d622953ee9fdd100aca98d9daff53e7e6d00e2b0" exitCode=0
Oct 31 00:42:32.687409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:32.687266    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q" event={"ID":"e86b5a81-4a0c-497c-9890-7ae79322144e","Type":"ContainerDied","Data":"8cf285a8e0b69b196a1daa56d622953ee9fdd100aca98d9daff53e7e6d00e2b0"}
Oct 31 00:42:33.825609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.825582    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:33.973606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.973531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x\" (UniqueName: \"kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x\") pod \"e86b5a81-4a0c-497c-9890-7ae79322144e\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") "
Oct 31 00:42:33.973606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.973600    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca\") pod \"e86b5a81-4a0c-497c-9890-7ae79322144e\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") "
Oct 31 00:42:33.973791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.973638    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca\") pod \"e86b5a81-4a0c-497c-9890-7ae79322144e\" (UID: \"e86b5a81-4a0c-497c-9890-7ae79322144e\") "
Oct 31 00:42:33.974065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.974043    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca" pod "e86b5a81-4a0c-497c-9890-7ae79322144e" (UID: "e86b5a81-4a0c-497c-9890-7ae79322144e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:33.975880 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.975843    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x" pod "e86b5a81-4a0c-497c-9890-7ae79322144e" (UID: "e86b5a81-4a0c-497c-9890-7ae79322144e"). InnerVolumeSpecName "kube-api-access-lp66x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:33.975999 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:33.975950    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca" pod "e86b5a81-4a0c-497c-9890-7ae79322144e" (UID: "e86b5a81-4a0c-497c-9890-7ae79322144e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:42:34.075253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:34.075227    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp66x\" (UniqueName: \"kubernetes.io/projected/e86b5a81-4a0c-497c-9890-7ae79322144e-kube-api-access-lp66x\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:34.075253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:34.075256    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:34.075405 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:34.075265    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b5a81-4a0c-497c-9890-7ae79322144e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:34.698277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:34.698241    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q" event={"ID":"e86b5a81-4a0c-497c-9890-7ae79322144e","Type":"ContainerDied","Data":"239a3a3b27738b47cbc945486f5206f58255bed857e3e6e9cfaad29d646a31c4"}
Oct 31 00:42:34.698277 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:34.698263    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"
Oct 31 00:42:34.698530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:34.698324    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="239a3a3b27738b47cbc945486f5206f58255bed857e3e6e9cfaad29d646a31c4"
Oct 31 00:42:36.385362 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:42:36.385322    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:42:51.787502 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:51.787449    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"]
Oct 31 00:42:51.788019 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:51.787778    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" containerID="cri-o://a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6" gracePeriod=30
Oct 31 00:42:51.790307 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:51.790281    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"]
Oct 31 00:42:51.790732 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:51.790661    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" containerID="cri-o://c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706" gracePeriod=30
Oct 31 00:42:51.792482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:51.792447    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"]
Oct 31 00:42:51.792800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:51.792771    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" containerID="cri-o://0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a" gracePeriod=30
Oct 31 00:42:52.796213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.796177    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"]
Oct 31 00:42:52.797000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.796965    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e86b5a81-4a0c-497c-9890-7ae79322144e" containerName="revoke-keys"
Oct 31 00:42:52.797000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.796991    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86b5a81-4a0c-497c-9890-7ae79322144e" containerName="revoke-keys"
Oct 31 00:42:52.797144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.797124    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86b5a81-4a0c-497c-9890-7ae79322144e" containerName="revoke-keys"
Oct 31 00:42:52.807900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.807875    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.816207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.816181    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"]
Oct 31 00:42:52.819838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.819816    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3a9c50-openshift-default-dockercfg-4ffhr\""
Oct 31 00:42:52.867885 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.867857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868007 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.867913    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n75\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.867998    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.868066    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.868092    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.868115    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.868176    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.868208    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.868293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.868267    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.969869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.969826    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n75\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.969892    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.969934    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.969970    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.970012    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.970072    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.970116    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.970145    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.970443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.970197    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.976260    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.976315    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.976973    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.977001    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.977321    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.977332    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.983527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.978945    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.988449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.988385    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:52.991321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:52.991259    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n75\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75\") pod \"e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") " pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:53.128283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.128201    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:53.441357 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.441335    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"]
Oct 31 00:42:53.443221 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:42:53.443191    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aefd9dd_1e76_418b_9de8_22c505a8aa1b.slice/crio-54cdbb5cd226eecdef35cfdf83c938e2c13ef329e348ed19566a75e879db35bd WatchSource:0}: Error finding container 54cdbb5cd226eecdef35cfdf83c938e2c13ef329e348ed19566a75e879db35bd: Status 404 returned error can't find the container with id 54cdbb5cd226eecdef35cfdf83c938e2c13ef329e348ed19566a75e879db35bd
Oct 31 00:42:53.445645 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.445614    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:42:53.445731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.445677    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:42:53.445731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.445706    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:42:53.817068 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.816974    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" event={"ID":"3aefd9dd-1e76-418b-9de8-22c505a8aa1b","Type":"ContainerStarted","Data":"1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"}
Oct 31 00:42:53.817589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.817076    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" event={"ID":"3aefd9dd-1e76-418b-9de8-22c505a8aa1b","Type":"ContainerStarted","Data":"54cdbb5cd226eecdef35cfdf83c938e2c13ef329e348ed19566a75e879db35bd"}
Oct 31 00:42:53.848877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:53.848828    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podStartSLOduration=1.848812586 podStartE2EDuration="1.848812586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:53.846456449 +0000 UTC m=+2941.792827172" watchObservedRunningTime="2026-10-31 00:42:53.848812586 +0000 UTC m=+2941.795183305"
Oct 31 00:42:54.129447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:54.129353    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:54.134857 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:54.134834    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:54.822702 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:54.822662    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:54.823739 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:54.823719    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:42:55.307776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:55.307738    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:42:55.307978 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:55.307801    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:42:55.330743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:55.330712    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-fstrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:42:55.330869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:55.330762    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:42:55.434893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:55.434862    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5x24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:42:55.435063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:55.434910    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:42:57.060476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.060432    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:42:57.104297 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.104275    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:42:57.107572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.107546    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:42:57.218808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218774    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.219000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218827    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.219000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218868    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.219000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218886    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.219000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218913    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.219000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218938    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.219000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.218974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219006    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219071    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219101    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219141    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219139    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219172    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.219250 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219206    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219303    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219389    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219567    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219617    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219617    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219637    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219665    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219689    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219751    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219776    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219807    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219840    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219876    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9\") pod \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\" (UID: \"47f1a6d4-f73a-437d-8f92-5e027d6f0f96\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219907    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy\") pod \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\" (UID: \"5bfb2dac-d2ee-49e2-8c34-71bb157b0885\") "
Oct 31 00:42:57.220289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219930    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token\") pod \"0c1437fb-5c91-492e-ae01-54983042358a\" (UID: \"0c1437fb-5c91-492e-ae01-54983042358a\") "
Oct 31 00:42:57.221557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.219774    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.221557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.220333    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.221557 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.220874    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221844    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221873    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221889    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221904    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221918    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221932    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222004 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.221948    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.222730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.222367    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.222730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.222496    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.222730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.222687    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.223572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.223241    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:57.223572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.223254    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.223572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.223379    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.223572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.223489    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:57.223908 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.223617    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:42:57.224255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.224230    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.224621 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.224585    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.225630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.225595    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:42:57.227132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.227106    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.227258 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.227205    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "kube-api-access-l4zn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:57.227451 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.227421    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "kube-api-access-xkwbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:57.227588 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.227566    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "kube-api-access-5xjzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:57.227684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.227602    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token" pod "47f1a6d4-f73a-437d-8f92-5e027d6f0f96" (UID: "47f1a6d4-f73a-437d-8f92-5e027d6f0f96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:57.227733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.227710    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:42:57.228086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.228059    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token" pod "0c1437fb-5c91-492e-ae01-54983042358a" (UID: "0c1437fb-5c91-492e-ae01-54983042358a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:57.228734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.228713    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:57.228734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.228729    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token" pod "5bfb2dac-d2ee-49e2-8c34-71bb157b0885" (UID: "5bfb2dac-d2ee-49e2-8c34-71bb157b0885"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322849    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322874    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322884    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322893    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322902    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322910    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1437fb-5c91-492e-ae01-54983042358a-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322918    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwbg\" (UniqueName: \"kubernetes.io/projected/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-kube-api-access-xkwbg\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.322923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322928    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322938    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xjzq\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-kube-api-access-5xjzq\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322946    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322954    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322962    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1437fb-5c91-492e-ae01-54983042358a-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322970    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zn9\" (UniqueName: \"kubernetes.io/projected/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-kube-api-access-l4zn9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322979    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322987    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1437fb-5c91-492e-ae01-54983042358a-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.322995    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.323003    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.323013    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfb2dac-d2ee-49e2-8c34-71bb157b0885-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.323021    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f1a6d4-f73a-437d-8f92-5e027d6f0f96-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.323452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.323029    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1437fb-5c91-492e-ae01-54983042358a-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:57.839803 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.839769    5078 generic.go:356] "Generic (PLEG): container finished" podID="0c1437fb-5c91-492e-ae01-54983042358a" containerID="c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706" exitCode=0
Oct 31 00:42:57.839995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.839828    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"
Oct 31 00:42:57.839995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.839860    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" event={"ID":"0c1437fb-5c91-492e-ae01-54983042358a","Type":"ContainerDied","Data":"c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706"}
Oct 31 00:42:57.839995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.839934    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b" event={"ID":"0c1437fb-5c91-492e-ae01-54983042358a","Type":"ContainerDied","Data":"d578b9cad7fe3136e86eb3048088f9cee234cc945d88d2720255c61ce0ff1827"}
Oct 31 00:42:57.839995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.839968    5078 scope.go:122] "RemoveContainer" containerID="c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706"
Oct 31 00:42:57.841359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.841327    5078 generic.go:356] "Generic (PLEG): container finished" podID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerID="0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a" exitCode=0
Oct 31 00:42:57.841475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.841394    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"
Oct 31 00:42:57.841475 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.841411    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" event={"ID":"47f1a6d4-f73a-437d-8f92-5e027d6f0f96","Type":"ContainerDied","Data":"0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a"}
Oct 31 00:42:57.841594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.841456    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj" event={"ID":"47f1a6d4-f73a-437d-8f92-5e027d6f0f96","Type":"ContainerDied","Data":"ae9abee5d9d09c9b8f4d73e220869ed0d05756c456c4f45fee4a345d55446be1"}
Oct 31 00:42:57.844044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.844020    5078 generic.go:356] "Generic (PLEG): container finished" podID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerID="a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6" exitCode=0
Oct 31 00:42:57.844191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.844063    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" event={"ID":"5bfb2dac-d2ee-49e2-8c34-71bb157b0885","Type":"ContainerDied","Data":"a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6"}
Oct 31 00:42:57.844191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.844091    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v" event={"ID":"5bfb2dac-d2ee-49e2-8c34-71bb157b0885","Type":"ContainerDied","Data":"88953ccaafa6d0d6998a4b1745626547ef417810b003affc95b5f4a053acb83e"}
Oct 31 00:42:57.844191 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.844144    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"
Oct 31 00:42:57.855520 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.855502    5078 scope.go:122] "RemoveContainer" containerID="c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706"
Oct 31 00:42:57.856290 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:42:57.856269    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706\": container with ID starting with c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706 not found: ID does not exist" containerID="c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706"
Oct 31 00:42:57.856383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.856300    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706"} err="failed to get container status \"c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706\": rpc error: code = NotFound desc = could not find container \"c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706\": container with ID starting with c4355f4b1ff320890f8a13d0816589f50deeef628c5b611152c5d2f886e52706 not found: ID does not exist"
Oct 31 00:42:57.856383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.856317    5078 scope.go:122] "RemoveContainer" containerID="0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a"
Oct 31 00:42:57.874457 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.874438    5078 scope.go:122] "RemoveContainer" containerID="0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a"
Oct 31 00:42:57.874770 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:42:57.874752    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a\": container with ID starting with 0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a not found: ID does not exist" containerID="0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a"
Oct 31 00:42:57.874840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.874778    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a"} err="failed to get container status \"0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a\": rpc error: code = NotFound desc = could not find container \"0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a\": container with ID starting with 0cf73a8e6f0b07e1b05514cffe6f3a59c3866d9b8949892df42b5b2a2810a61a not found: ID does not exist"
Oct 31 00:42:57.874840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.874794    5078 scope.go:122] "RemoveContainer" containerID="a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6"
Oct 31 00:42:57.883117 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.882986    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"]
Oct 31 00:42:57.885116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.885088    5078 scope.go:122] "RemoveContainer" containerID="a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6"
Oct 31 00:42:57.885490 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:42:57.885443    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6\": container with ID starting with a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6 not found: ID does not exist" containerID="a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6"
Oct 31 00:42:57.885612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.885590    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6"} err="failed to get container status \"a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6\": rpc error: code = NotFound desc = could not find container \"a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6\": container with ID starting with a423c253c2d2f9c7006016f4192b6b254c16407aa61e71a154ae87d50b1a3ac6 not found: ID does not exist"
Oct 31 00:42:57.885872 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.885856    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5x24b"]
Oct 31 00:42:57.900226 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.900200    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"]
Oct 31 00:42:57.908090 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.908068    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fstrj"]
Oct 31 00:42:57.919550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.919529    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"]
Oct 31 00:42:57.924812 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:57.924793    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-f8j9v"]
Oct 31 00:42:58.192921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:58.192882    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1437fb-5c91-492e-ae01-54983042358a" path="/var/lib/kubelet/pods/0c1437fb-5c91-492e-ae01-54983042358a/volumes"
Oct 31 00:42:58.193583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:58.193561    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" path="/var/lib/kubelet/pods/47f1a6d4-f73a-437d-8f92-5e027d6f0f96/volumes"
Oct 31 00:42:58.194594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:42:58.194574    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" path="/var/lib/kubelet/pods/5bfb2dac-d2ee-49e2-8c34-71bb157b0885/volumes"
Oct 31 00:43:06.404325 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:43:06.404289    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:43:06.953820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.953789    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"]
Oct 31 00:43:06.954521 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954500    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy"
Oct 31 00:43:06.954577 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954525    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy"
Oct 31 00:43:06.954620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954578    5078 cpu_manager.go:414] "containerMap: removing container" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy"
Oct 31 00:43:06.954620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954588    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy"
Oct 31 00:43:06.954620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954608    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy"
Oct 31 00:43:06.954724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954619    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy"
Oct 31 00:43:06.954759 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954736    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfb2dac-d2ee-49e2-8c34-71bb157b0885" containerName="istio-proxy"
Oct 31 00:43:06.954790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954757    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f1a6d4-f73a-437d-8f92-5e027d6f0f96" containerName="istio-proxy"
Oct 31 00:43:06.954790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.954767    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1437fb-5c91-492e-ae01-54983042358a" containerName="istio-proxy"
Oct 31 00:43:06.960095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.960071    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:06.964544 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.964522    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"]
Oct 31 00:43:06.968124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.968104    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:06.985500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.981858    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"]
Oct 31 00:43:06.988264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:06.988235    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"]
Oct 31 00:43:07.017855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.017832    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p7q4d\""
Oct 31 00:43:07.037040 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037010    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtj5f\" (UniqueName: \"kubernetes.io/projected/91ec5a7e-3f44-42e3-907f-c431831b4065-kube-api-access-qtj5f\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037047    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ec5a7e-3f44-42e3-907f-c431831b4065-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037079    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037150    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2f60f1-58ff-4202-a48a-fb057254c182-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037185    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037349    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037409    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037444    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt62g\" (UniqueName: \"kubernetes.io/projected/eb2f60f1-58ff-4202-a48a-fb057254c182-kube-api-access-qt62g\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037504    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037575 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037537    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037618    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037646    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037680    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037743 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037725    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.037919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037791    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037852    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.037919 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.037898    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.139070 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139037    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtj5f\" (UniqueName: \"kubernetes.io/projected/91ec5a7e-3f44-42e3-907f-c431831b4065-kube-api-access-qtj5f\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.139260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139084    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ec5a7e-3f44-42e3-907f-c431831b4065-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.139260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139128    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.139260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139155    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2f60f1-58ff-4202-a48a-fb057254c182-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.139260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139185    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.139260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139225    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.139260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139257    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.139642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139281    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.139642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139503    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt62g\" (UniqueName: \"kubernetes.io/projected/eb2f60f1-58ff-4202-a48a-fb057254c182-kube-api-access-qt62g\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.139642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139559    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.139642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139597    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.139642 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139635    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139671    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139676    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139749    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139783    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139814    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139858    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139905    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139939    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139961    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ec5a7e-3f44-42e3-907f-c431831b4065-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.139980    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2f60f1-58ff-4202-a48a-fb057254c182-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.140660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.140209    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.140369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.140479    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.140660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.140506    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.140891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.140854    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.141285 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.141261    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.141868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.141841    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.142014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.141961    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.142710 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.142688    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.142889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.142875    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.156839 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.156813    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2f60f1-58ff-4202-a48a-fb057254c182-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.156966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.156843    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt62g\" (UniqueName: \"kubernetes.io/projected/eb2f60f1-58ff-4202-a48a-fb057254c182-kube-api-access-qt62g\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6plnx\" (UID: \"eb2f60f1-58ff-4202-a48a-fb057254c182\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.157945 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.157924    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ec5a7e-3f44-42e3-907f-c431831b4065-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.161709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.161688    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtj5f\" (UniqueName: \"kubernetes.io/projected/91ec5a7e-3f44-42e3-907f-c431831b4065-kube-api-access-qtj5f\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-mr9zl\" (UID: \"91ec5a7e-3f44-42e3-907f-c431831b4065\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.283582 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.283495    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:07.299811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.299624    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:07.573608 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.573580    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"]
Oct 31 00:43:07.574681 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:43:07.574644    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2f60f1_58ff_4202_a48a_fb057254c182.slice/crio-4522bc701c0c6a5a9397681cd17afc080e1776c0495318bc798923dfd66c54f9 WatchSource:0}: Error finding container 4522bc701c0c6a5a9397681cd17afc080e1776c0495318bc798923dfd66c54f9: Status 404 returned error can't find the container with id 4522bc701c0c6a5a9397681cd17afc080e1776c0495318bc798923dfd66c54f9
Oct 31 00:43:07.577223 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.577196    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:07.577314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.577259    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:07.577314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.577286    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:07.596333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.596309    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"]
Oct 31 00:43:07.605862 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:43:07.605831    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ec5a7e_3f44_42e3_907f_c431831b4065.slice/crio-4ef52511351e3c15cc51ef5ebf441aaa697633eea3a4193ae7adbd28fc7f449e WatchSource:0}: Error finding container 4ef52511351e3c15cc51ef5ebf441aaa697633eea3a4193ae7adbd28fc7f449e: Status 404 returned error can't find the container with id 4ef52511351e3c15cc51ef5ebf441aaa697633eea3a4193ae7adbd28fc7f449e
Oct 31 00:43:07.608420 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.608388    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:07.608533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.608484    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:07.608533 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.608525    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:07.932924 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.932880    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" event={"ID":"eb2f60f1-58ff-4202-a48a-fb057254c182","Type":"ContainerStarted","Data":"bae89acec6292c34b9366db8be74dc671eabe0e8795b8cae86179520f6dfc8a1"}
Oct 31 00:43:07.933125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.932949    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" event={"ID":"eb2f60f1-58ff-4202-a48a-fb057254c182","Type":"ContainerStarted","Data":"4522bc701c0c6a5a9397681cd17afc080e1776c0495318bc798923dfd66c54f9"}
Oct 31 00:43:07.935369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.935333    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" event={"ID":"91ec5a7e-3f44-42e3-907f-c431831b4065","Type":"ContainerStarted","Data":"d57e431f77d0e37d0f1448e836109c0e8fbe9db2579894723d9da050b51448f4"}
Oct 31 00:43:07.935537 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.935393    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" event={"ID":"91ec5a7e-3f44-42e3-907f-c431831b4065","Type":"ContainerStarted","Data":"4ef52511351e3c15cc51ef5ebf441aaa697633eea3a4193ae7adbd28fc7f449e"}
Oct 31 00:43:07.963637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:07.963575    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podStartSLOduration=1.963555776 podStartE2EDuration="1.963555776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.961911079 +0000 UTC m=+2955.908281798" watchObservedRunningTime="2026-10-31 00:43:07.963555776 +0000 UTC m=+2955.909926493"
Oct 31 00:43:08.292903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:08.292767    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:08.294838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:08.294806    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:08.295063 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:08.295036    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:08.306486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:08.304289    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:08.314486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:08.313453    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:08.314486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:08.313582    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:09.289490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:09.288390    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:09.289490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:09.288450    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:09.302891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:09.302787    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:09.302891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:09.302838    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:10.290487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:10.286877    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:10.290487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:10.286940    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:10.300830 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:10.300797    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:10.301075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:10.301048    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:10.864631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:10.864595    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:43:10.864843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:10.864651    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:43:11.292325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:11.291878    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:11.292325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:11.291940    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:11.301545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:11.301414    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:11.301545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:11.301491    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:12.294491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.293578    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:12.294491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.293640    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:12.304490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.301792    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:12.304490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.301846    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:12.966333 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.966294    5078 generic.go:356] "Generic (PLEG): container finished" podID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerID="1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025" exitCode=0
Oct 31 00:43:12.966536 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.966421    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" event={"ID":"3aefd9dd-1e76-418b-9de8-22c505a8aa1b","Type":"ContainerDied","Data":"1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"}
Oct 31 00:43:12.966883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.966868    5078 scope.go:122] "RemoveContainer" containerID="1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"
Oct 31 00:43:12.968045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.968018    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:12.968109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.968096    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:12.968155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:12.968144    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:43:13.024322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:13.024246    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podStartSLOduration=7.024225541 podStartE2EDuration="7.024225541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.99166262 +0000 UTC m=+2955.938033343" watchObservedRunningTime="2026-10-31 00:43:13.024225541 +0000 UTC m=+2960.970596260"
Oct 31 00:43:13.294548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:13.288843    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:13.294548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:13.288904    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:13.326498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:13.310833    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:13.326498 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:13.310892    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:13.995488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:13.988062    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" event={"ID":"3aefd9dd-1e76-418b-9de8-22c505a8aa1b","Type":"ContainerStarted","Data":"63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa"}
Oct 31 00:43:14.128640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.128605    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:43:14.128882 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.128854    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 31 00:43:14.128938 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.128918    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 31 00:43:14.285548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.284858    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:14.285548 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.284921    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:14.303678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.303559    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:14.303678 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:14.303619    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:15.129199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:15.129152    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 31 00:43:15.129429 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:15.129210    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 31 00:43:15.290488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:15.289962    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 00:43:15.290488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:15.290047    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 00:43:15.314631 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:15.314595    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 00:43:15.315166 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:15.314672    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 00:43:16.131488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.130980    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 31 00:43:16.131488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.131104    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 31 00:43:16.890489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.886823    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:16.890489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.888292    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:16.896772 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.896746    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx"
Oct 31 00:43:16.997487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.993889    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:16.997487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:16.995523    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:17.003487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:17.001670    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl"
Oct 31 00:43:17.129289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:17.128747    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 31 00:43:17.129289 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:17.128812    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 31 00:43:18.129453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:18.129405    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 31 00:43:18.129981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:18.129509    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 31 00:43:19.129069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:19.129037    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 31 00:43:19.129255 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:19.129101    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 31 00:43:20.133270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:20.133240    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:43:20.133806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:20.133673    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:43:20.134395 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:20.134374    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:43:32.190733 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:32.190697    5078 scope.go:122] "RemoveContainer" containerID="13c9e0fc979591e4f7a3ee3c27de0d9042ea07cb51e9c865973f5642938cff44"
Oct 31 00:43:32.210118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:32.210093    5078 scope.go:122] "RemoveContainer" containerID="ec9a38875d59e1a42fb2b83973f1ec86d1912a759ed924ea19e04ea8d9ce2e60"
Oct 31 00:43:36.385764 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:43:36.385727    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:43:44.306818 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.306788    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"]
Oct 31 00:43:44.311948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.311930    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.317683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.317657    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:43:44.317683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.317666    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:43:44.321625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.321608    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:43:44.327083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.327055    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"]
Oct 31 00:43:44.418328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.418298    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:43:44.458772 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.458740    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9td9c\" (UniqueName: \"kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.458913 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.458787    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.458957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.458923    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.560241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.560173    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.560241 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.560236    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9td9c\" (UniqueName: \"kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.560410 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.560263    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.560928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.560908    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.562765 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.562742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.577926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.577904    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9td9c\" (UniqueName: \"kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c\") pod \"maas-api-revoke-sub-9d1f03d5-rtrfh\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.625088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.625065    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:44.910325 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:44.910297    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"]
Oct 31 00:43:44.912167 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:43:44.912127    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846fbd26_e8b3_4725_9ccd_58d96d07b035.slice/crio-7415be3bf06ed44b44107d84cbeb7c9fadfa75711f4acbd1a87869ee40492aa0 WatchSource:0}: Error finding container 7415be3bf06ed44b44107d84cbeb7c9fadfa75711f4acbd1a87869ee40492aa0: Status 404 returned error can't find the container with id 7415be3bf06ed44b44107d84cbeb7c9fadfa75711f4acbd1a87869ee40492aa0
Oct 31 00:43:45.204402 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:45.204376    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh" event={"ID":"846fbd26-e8b3-4725-9ccd-58d96d07b035","Type":"ContainerStarted","Data":"5fa2690a3ed13f20d6eff4ff56fda3abf83c97a3bc22d09a7edb4b834a12b403"}
Oct 31 00:43:45.204534 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:45.204417    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh" event={"ID":"846fbd26-e8b3-4725-9ccd-58d96d07b035","Type":"ContainerStarted","Data":"7415be3bf06ed44b44107d84cbeb7c9fadfa75711f4acbd1a87869ee40492aa0"}
Oct 31 00:43:45.226227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:45.226181    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh" podStartSLOduration=1.226169245 podStartE2EDuration="1.226169245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:45.224358188 +0000 UTC m=+2993.170728906" watchObservedRunningTime="2026-10-31 00:43:45.226169245 +0000 UTC m=+2993.172539956"
Oct 31 00:43:46.211132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:46.211097    5078 generic.go:356] "Generic (PLEG): container finished" podID="846fbd26-e8b3-4725-9ccd-58d96d07b035" containerID="5fa2690a3ed13f20d6eff4ff56fda3abf83c97a3bc22d09a7edb4b834a12b403" exitCode=0
Oct 31 00:43:46.211132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:46.211132    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh" event={"ID":"846fbd26-e8b3-4725-9ccd-58d96d07b035","Type":"ContainerDied","Data":"5fa2690a3ed13f20d6eff4ff56fda3abf83c97a3bc22d09a7edb4b834a12b403"}
Oct 31 00:43:47.359568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.359540    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:47.490360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.490268    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca\") pod \"846fbd26-e8b3-4725-9ccd-58d96d07b035\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") "
Oct 31 00:43:47.490583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.490380    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca\") pod \"846fbd26-e8b3-4725-9ccd-58d96d07b035\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") "
Oct 31 00:43:47.490583 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.490424    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c\" (UniqueName: \"kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c\") pod \"846fbd26-e8b3-4725-9ccd-58d96d07b035\" (UID: \"846fbd26-e8b3-4725-9ccd-58d96d07b035\") "
Oct 31 00:43:47.490711 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.490650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca" pod "846fbd26-e8b3-4725-9ccd-58d96d07b035" (UID: "846fbd26-e8b3-4725-9ccd-58d96d07b035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:47.490811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.490794    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:47.492677 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.492653    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca" pod "846fbd26-e8b3-4725-9ccd-58d96d07b035" (UID: "846fbd26-e8b3-4725-9ccd-58d96d07b035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:43:47.492757 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.492736    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c" pod "846fbd26-e8b3-4725-9ccd-58d96d07b035" (UID: "846fbd26-e8b3-4725-9ccd-58d96d07b035"). InnerVolumeSpecName "kube-api-access-9td9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:47.592222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.592188    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846fbd26-e8b3-4725-9ccd-58d96d07b035-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:47.592222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:47.592219    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9td9c\" (UniqueName: \"kubernetes.io/projected/846fbd26-e8b3-4725-9ccd-58d96d07b035-kube-api-access-9td9c\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:48.222328 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:48.222299    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"
Oct 31 00:43:48.222508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:48.222332    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh" event={"ID":"846fbd26-e8b3-4725-9ccd-58d96d07b035","Type":"ContainerDied","Data":"7415be3bf06ed44b44107d84cbeb7c9fadfa75711f4acbd1a87869ee40492aa0"}
Oct 31 00:43:48.222508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:48.222385    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7415be3bf06ed44b44107d84cbeb7c9fadfa75711f4acbd1a87869ee40492aa0"
Oct 31 00:43:54.569119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:54.569093    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:43:54.572606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:54.572586    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:43:54.574741 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:54.574722    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:43:54.578075 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:54.578058    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:43:55.525640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.525610    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"]
Oct 31 00:43:55.526072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.526059    5078 cpu_manager.go:414] "containerMap: removing container" podUID="846fbd26-e8b3-4725-9ccd-58d96d07b035" containerName="revoke-keys"
Oct 31 00:43:55.526121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.526073    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846fbd26-e8b3-4725-9ccd-58d96d07b035" containerName="revoke-keys"
Oct 31 00:43:55.526158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.526137    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="846fbd26-e8b3-4725-9ccd-58d96d07b035" containerName="revoke-keys"
Oct 31 00:43:55.529264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.529244    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.538862 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.538843    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:43:55.539006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.538913    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:43:55.539101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.539087    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:43:55.539178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.539160    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:43:55.558721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.558696    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"]
Oct 31 00:43:55.566799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.566763    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.566915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.566856    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt49g\" (UniqueName: \"kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.567000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.566983    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.667669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.667637    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.668125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.667683    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.668125 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.667722    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt49g\" (UniqueName: \"kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.668515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.668457    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.670182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.670163    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.682101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.682077    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt49g\" (UniqueName: \"kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:55.842820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:55.842733    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:56.136012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:56.135976    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"]
Oct 31 00:43:56.136305 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:43:56.136273    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa187714_160c_40bb_8738_4d3d339d4526.slice/crio-04263b8813a2f828c7e9d22aa8105e648fb1aa32e1457c22367ce8953b8333f4 WatchSource:0}: Error finding container 04263b8813a2f828c7e9d22aa8105e648fb1aa32e1457c22367ce8953b8333f4: Status 404 returned error can't find the container with id 04263b8813a2f828c7e9d22aa8105e648fb1aa32e1457c22367ce8953b8333f4
Oct 31 00:43:56.262850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:56.262815    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g" event={"ID":"fa187714-160c-40bb-8738-4d3d339d4526","Type":"ContainerStarted","Data":"04263b8813a2f828c7e9d22aa8105e648fb1aa32e1457c22367ce8953b8333f4"}
Oct 31 00:43:57.272360 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:57.272325    5078 generic.go:356] "Generic (PLEG): container finished" podID="fa187714-160c-40bb-8738-4d3d339d4526" containerID="8855f164e343caef42bbbc1942525495adc1ce4ba5bd6dd15ade4843e9d0ef75" exitCode=0
Oct 31 00:43:57.272832 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:57.272431    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g" event={"ID":"fa187714-160c-40bb-8738-4d3d339d4526","Type":"ContainerDied","Data":"8855f164e343caef42bbbc1942525495adc1ce4ba5bd6dd15ade4843e9d0ef75"}
Oct 31 00:43:58.423661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.423631    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:58.501115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.501081    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca\") pod \"fa187714-160c-40bb-8738-4d3d339d4526\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") "
Oct 31 00:43:58.501295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.501152    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g\" (UniqueName: \"kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g\") pod \"fa187714-160c-40bb-8738-4d3d339d4526\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") "
Oct 31 00:43:58.501295 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.501269    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca\") pod \"fa187714-160c-40bb-8738-4d3d339d4526\" (UID: \"fa187714-160c-40bb-8738-4d3d339d4526\") "
Oct 31 00:43:58.501550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.501522    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca" pod "fa187714-160c-40bb-8738-4d3d339d4526" (UID: "fa187714-160c-40bb-8738-4d3d339d4526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:58.501672 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.501652    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:58.503497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.503476    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca" pod "fa187714-160c-40bb-8738-4d3d339d4526" (UID: "fa187714-160c-40bb-8738-4d3d339d4526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:43:58.503604 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.503590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g" pod "fa187714-160c-40bb-8738-4d3d339d4526" (UID: "fa187714-160c-40bb-8738-4d3d339d4526"). InnerVolumeSpecName "kube-api-access-zt49g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:58.602903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.602840    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt49g\" (UniqueName: \"kubernetes.io/projected/fa187714-160c-40bb-8738-4d3d339d4526-kube-api-access-zt49g\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:58.602903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:58.602863    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa187714-160c-40bb-8738-4d3d339d4526-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:59.283605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:59.283571    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"
Oct 31 00:43:59.283787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:59.283571    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g" event={"ID":"fa187714-160c-40bb-8738-4d3d339d4526","Type":"ContainerDied","Data":"04263b8813a2f828c7e9d22aa8105e648fb1aa32e1457c22367ce8953b8333f4"}
Oct 31 00:43:59.283787 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:43:59.283704    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04263b8813a2f828c7e9d22aa8105e648fb1aa32e1457c22367ce8953b8333f4"
Oct 31 00:44:06.401510 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:44:06.401416    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:44:06.715276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:06.715192    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"]
Oct 31 00:44:06.728421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:06.728393    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a9c50-q299g"]
Oct 31 00:44:08.194946 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:08.194905    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa187714-160c-40bb-8738-4d3d339d4526" path="/var/lib/kubelet/pods/fa187714-160c-40bb-8738-4d3d339d4526/volumes"
Oct 31 00:44:13.955181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:13.955141    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"]
Oct 31 00:44:13.955814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:13.955439    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy" containerID="cri-o://63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa" gracePeriod=30
Oct 31 00:44:15.355917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.355884    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"]
Oct 31 00:44:15.359332 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.359306    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsp2q"]
Oct 31 00:44:15.385422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.385394    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"]
Oct 31 00:44:15.385894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.385879    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fa187714-160c-40bb-8738-4d3d339d4526" containerName="revoke-keys"
Oct 31 00:44:15.385894 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.385894    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa187714-160c-40bb-8738-4d3d339d4526" containerName="revoke-keys"
Oct 31 00:44:15.386043 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.385975    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa187714-160c-40bb-8738-4d3d339d4526" containerName="revoke-keys"
Oct 31 00:44:15.390620 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.390604    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.396441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.395984    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:44:15.396441 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.396059    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:44:15.399210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.399189    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:44:15.399359 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.399336    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:44:15.400158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.400140    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"]
Oct 31 00:44:15.492052 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.492013    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.492236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.492206    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67hc\" (UniqueName: \"kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.492283 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.492248    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.593655 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.593611    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67hc\" (UniqueName: \"kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.593847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.593675    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.593847 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.593747    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.594585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.594554    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.598774 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.598743    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.610445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.610364    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67hc\" (UniqueName: \"kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc\") pod \"maas-api-revoke-sub-3d4156f1-5xbv7\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.706630 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.706596    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:15.986783 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:15.986760    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"]
Oct 31 00:44:15.989548 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:44:15.989520    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c500154_30a5_478a_9eff_f94e43655527.slice/crio-41fcbb1e2ec69f5e2c8b4efb6c52aa7f41dc55c9dd76cbe805c57ec633a8d08c WatchSource:0}: Error finding container 41fcbb1e2ec69f5e2c8b4efb6c52aa7f41dc55c9dd76cbe805c57ec633a8d08c: Status 404 returned error can't find the container with id 41fcbb1e2ec69f5e2c8b4efb6c52aa7f41dc55c9dd76cbe805c57ec633a8d08c
Oct 31 00:44:16.199560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:16.199526    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86b5a81-4a0c-497c-9890-7ae79322144e" path="/var/lib/kubelet/pods/e86b5a81-4a0c-497c-9890-7ae79322144e/volumes"
Oct 31 00:44:16.378822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:16.378787    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7" event={"ID":"7c500154-30a5-478a-9eff-f94e43655527","Type":"ContainerStarted","Data":"044b11a202aa754ed40db4d385a1530f7a3d9390b7120f579c5a135affe2caaf"}
Oct 31 00:44:16.379168 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:16.378844    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7" event={"ID":"7c500154-30a5-478a-9eff-f94e43655527","Type":"ContainerStarted","Data":"41fcbb1e2ec69f5e2c8b4efb6c52aa7f41dc55c9dd76cbe805c57ec633a8d08c"}
Oct 31 00:44:16.400658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:16.400613    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7" podStartSLOduration=1.400600621 podStartE2EDuration="1.400600621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:16.399567789 +0000 UTC m=+3024.345938513" watchObservedRunningTime="2026-10-31 00:44:16.400600621 +0000 UTC m=+3024.346971331"
Oct 31 00:44:17.386132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:17.386100    5078 generic.go:356] "Generic (PLEG): container finished" podID="7c500154-30a5-478a-9eff-f94e43655527" containerID="044b11a202aa754ed40db4d385a1530f7a3d9390b7120f579c5a135affe2caaf" exitCode=0
Oct 31 00:44:17.386563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:17.386188    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7" event={"ID":"7c500154-30a5-478a-9eff-f94e43655527","Type":"ContainerDied","Data":"044b11a202aa754ed40db4d385a1530f7a3d9390b7120f579c5a135affe2caaf"}
Oct 31 00:44:18.544309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.544284    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:18.637074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.637045    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca\") pod \"7c500154-30a5-478a-9eff-f94e43655527\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") "
Oct 31 00:44:18.637235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.637100    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc\" (UniqueName: \"kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc\") pod \"7c500154-30a5-478a-9eff-f94e43655527\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") "
Oct 31 00:44:18.637235 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.637206    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca\") pod \"7c500154-30a5-478a-9eff-f94e43655527\" (UID: \"7c500154-30a5-478a-9eff-f94e43655527\") "
Oct 31 00:44:18.637590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.637567    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca" pod "7c500154-30a5-478a-9eff-f94e43655527" (UID: "7c500154-30a5-478a-9eff-f94e43655527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:44:18.639318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.639293    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc" pod "7c500154-30a5-478a-9eff-f94e43655527" (UID: "7c500154-30a5-478a-9eff-f94e43655527"). InnerVolumeSpecName "kube-api-access-t67hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:18.639318 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.639297    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca" pod "7c500154-30a5-478a-9eff-f94e43655527" (UID: "7c500154-30a5-478a-9eff-f94e43655527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:44:18.738242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.738217    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c500154-30a5-478a-9eff-f94e43655527-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:18.738242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.738241    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67hc\" (UniqueName: \"kubernetes.io/projected/7c500154-30a5-478a-9eff-f94e43655527-kube-api-access-t67hc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:18.738376 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:18.738250    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c500154-30a5-478a-9eff-f94e43655527-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.200618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.200594    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:44:19.241781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241718    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241775    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241865    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241885    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241935    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241953    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.241974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.241998 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.242002    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.242353 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.242290    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:19.242840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.242530    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:19.242840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.242613    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:19.242840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.242735    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:19.242840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.242827    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:44:19.244589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.244555    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:44:19.244696 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.244642    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "kube-api-access-w6n75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:19.244897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.244882    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:19.343107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343080    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy\") pod \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\" (UID: \"3aefd9dd-1e76-418b-9de8-22c505a8aa1b\") "
Oct 31 00:44:19.343361 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343348    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-data\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343364    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-credential-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343373    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-token\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343381    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-socket\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343390    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-podinfo\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343399    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6n75\" (UniqueName: \"kubernetes.io/projected/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-kube-api-access-w6n75\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343411 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343409    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istiod-ca-cert\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.343635 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.343418    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-workload-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:19.345112 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.345093    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy" pod "3aefd9dd-1e76-418b-9de8-22c505a8aa1b" (UID: "3aefd9dd-1e76-418b-9de8-22c505a8aa1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:19.396915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.396888    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"
Oct 31 00:44:19.397048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.396914    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7" event={"ID":"7c500154-30a5-478a-9eff-f94e43655527","Type":"ContainerDied","Data":"41fcbb1e2ec69f5e2c8b4efb6c52aa7f41dc55c9dd76cbe805c57ec633a8d08c"}
Oct 31 00:44:19.397048 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.396961    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41fcbb1e2ec69f5e2c8b4efb6c52aa7f41dc55c9dd76cbe805c57ec633a8d08c"
Oct 31 00:44:19.398693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.398669    5078 generic.go:356] "Generic (PLEG): container finished" podID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerID="63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa" exitCode=0
Oct 31 00:44:19.398823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.398716    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"
Oct 31 00:44:19.398823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.398719    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" event={"ID":"3aefd9dd-1e76-418b-9de8-22c505a8aa1b","Type":"ContainerDied","Data":"63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa"}
Oct 31 00:44:19.398971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.398837    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh" event={"ID":"3aefd9dd-1e76-418b-9de8-22c505a8aa1b","Type":"ContainerDied","Data":"54cdbb5cd226eecdef35cfdf83c938e2c13ef329e348ed19566a75e879db35bd"}
Oct 31 00:44:19.398971 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.398862    5078 scope.go:122] "RemoveContainer" containerID="63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa"
Oct 31 00:44:19.409087 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.409071    5078 scope.go:122] "RemoveContainer" containerID="1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"
Oct 31 00:44:19.417526 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.417505    5078 scope.go:122] "RemoveContainer" containerID="63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa"
Oct 31 00:44:19.417777 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:44:19.417756    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa\": container with ID starting with 63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa not found: ID does not exist" containerID="63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa"
Oct 31 00:44:19.417834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.417785    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa"} err="failed to get container status \"63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa\": rpc error: code = NotFound desc = could not find container \"63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa\": container with ID starting with 63e6ec82dc6d3658128332729c50ebcdec1756ac60aa166e4706825463ada3fa not found: ID does not exist"
Oct 31 00:44:19.417834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.417800    5078 scope.go:122] "RemoveContainer" containerID="1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"
Oct 31 00:44:19.418077 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:44:19.418058    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025\": container with ID starting with 1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025 not found: ID does not exist" containerID="1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"
Oct 31 00:44:19.418135 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.418084    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025"} err="failed to get container status \"1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025\": rpc error: code = NotFound desc = could not find container \"1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025\": container with ID starting with 1aa5ff29672f16f1ad819930cc6c38f69fcd2272b95b93d0e96958c93a837025 not found: ID does not exist"
Oct 31 00:44:19.433656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.433629    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"]
Oct 31 00:44:19.436132 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.436101    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a9c50-openshift-default-6bcb7f4cbfgxhkh"]
Oct 31 00:44:19.444558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:19.444537    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aefd9dd-1e76-418b-9de8-22c505a8aa1b-istio-envoy\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:20.192965 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:20.192929    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" path="/var/lib/kubelet/pods/3aefd9dd-1e76-418b-9de8-22c505a8aa1b/volumes"
Oct 31 00:44:36.385295 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:44:36.385253    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:44:37.392199 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.392168    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"]
Oct 31 00:44:37.392685 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.392374    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager" containerID="cri-o://38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634" gracePeriod=10
Oct 31 00:44:37.654045 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.653974    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:44:37.727000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.726974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume\") pod \"7e84018f-f7a2-4588-bd5b-b67814c509bd\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") "
Oct 31 00:44:37.727144 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.727026    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk\" (UniqueName: \"kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk\") pod \"7e84018f-f7a2-4588-bd5b-b67814c509bd\" (UID: \"7e84018f-f7a2-4588-bd5b-b67814c509bd\") "
Oct 31 00:44:37.727421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.727400    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume" pod "7e84018f-f7a2-4588-bd5b-b67814c509bd" (UID: "7e84018f-f7a2-4588-bd5b-b67814c509bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:37.729383 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.729354    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk" pod "7e84018f-f7a2-4588-bd5b-b67814c509bd" (UID: "7e84018f-f7a2-4588-bd5b-b67814c509bd"). InnerVolumeSpecName "kube-api-access-jcvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:37.828687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.828655    5078 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e84018f-f7a2-4588-bd5b-b67814c509bd-extensions-socket-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:37.828687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:37.828684    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvgk\" (UniqueName: \"kubernetes.io/projected/7e84018f-f7a2-4588-bd5b-b67814c509bd-kube-api-access-jcvgk\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:38.503426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.503337    5078 generic.go:356] "Generic (PLEG): container finished" podID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerID="38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634" exitCode=0
Oct 31 00:44:38.503426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.503383    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"
Oct 31 00:44:38.503935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.503418    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" event={"ID":"7e84018f-f7a2-4588-bd5b-b67814c509bd","Type":"ContainerDied","Data":"38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634"}
Oct 31 00:44:38.503935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.503489    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9" event={"ID":"7e84018f-f7a2-4588-bd5b-b67814c509bd","Type":"ContainerDied","Data":"d63e44546c3664762b8161c405dc36d3973989622271fa8218569681dbd6482b"}
Oct 31 00:44:38.503935 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.503514    5078 scope.go:122] "RemoveContainer" containerID="38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634"
Oct 31 00:44:38.514497 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.514480    5078 scope.go:122] "RemoveContainer" containerID="e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"
Oct 31 00:44:38.523122 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.523103    5078 scope.go:122] "RemoveContainer" containerID="38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634"
Oct 31 00:44:38.523401 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:44:38.523383    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634\": container with ID starting with 38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634 not found: ID does not exist" containerID="38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634"
Oct 31 00:44:38.523560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.523409    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634"} err="failed to get container status \"38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634\": rpc error: code = NotFound desc = could not find container \"38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634\": container with ID starting with 38a901fe802c0ec5315354aa7f850d9fe417e38c1ff1be29cd89751106e4c634 not found: ID does not exist"
Oct 31 00:44:38.523560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.523425    5078 scope.go:122] "RemoveContainer" containerID="e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"
Oct 31 00:44:38.523688 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:44:38.523666    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029\": container with ID starting with e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029 not found: ID does not exist" containerID="e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"
Oct 31 00:44:38.523688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.523683    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029"} err="failed to get container status \"e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029\": rpc error: code = NotFound desc = could not find container \"e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029\": container with ID starting with e6601e6fe409d59c1009f59b132e0df4f88e54070c5ef44b304560b5fd73a029 not found: ID does not exist"
Oct 31 00:44:38.527966 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.527945    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"]
Oct 31 00:44:38.532003 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:38.531985    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5xz9"]
Oct 31 00:44:40.193482 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:40.193426    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" path="/var/lib/kubelet/pods/7e84018f-f7a2-4588-bd5b-b67814c509bd/volumes"
Oct 31 00:44:52.060737 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.060700    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"]
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061157    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061175    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061204    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061212    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061231    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061239    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061251    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061255    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061269    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7c500154-30a5-478a-9eff-f94e43655527" containerName="revoke-keys"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061273    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c500154-30a5-478a-9eff-f94e43655527" containerName="revoke-keys"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061362    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061371    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84018f-f7a2-4588-bd5b-b67814c509bd" containerName="manager"
Oct 31 00:44:52.061369 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061379    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy"
Oct 31 00:44:52.062266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061386    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c500154-30a5-478a-9eff-f94e43655527" containerName="revoke-keys"
Oct 31 00:44:52.062266 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.061396    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aefd9dd-1e76-418b-9de8-22c505a8aa1b" containerName="istio-proxy"
Oct 31 00:44:52.064674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.064656    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.131716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.131689    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"]
Oct 31 00:44:52.154182 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154142    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154204    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154267    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154392 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154331    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154392    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f5e737-fb2d-4485-9501-c3f7f06146d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154426    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154517 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154475    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rxw\" (UniqueName: \"kubernetes.io/projected/98f5e737-fb2d-4485-9501-c3f7f06146d4-kube-api-access-v4rxw\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154563 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154547    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.154601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.154590    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.255797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.255762    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.255931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.255818    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.255931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.255852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.255931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.255894    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.255931 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.255921    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.255945    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256004    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f5e737-fb2d-4485-9501-c3f7f06146d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256053    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rxw\" (UniqueName: \"kubernetes.io/projected/98f5e737-fb2d-4485-9501-c3f7f06146d4-kube-api-access-v4rxw\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256194    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256358 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256286    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256492    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.256820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.256793    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f5e737-fb2d-4485-9501-c3f7f06146d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.258903 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.258878    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.259095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.259077    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.276097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.276079    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rxw\" (UniqueName: \"kubernetes.io/projected/98f5e737-fb2d-4485-9501-c3f7f06146d4-kube-api-access-v4rxw\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.276326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.276308    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f5e737-fb2d-4485-9501-c3f7f06146d4-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wxssj\" (UID: \"98f5e737-fb2d-4485-9501-c3f7f06146d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.380050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.379975    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:52.663787 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:44:52.663755    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f5e737_fb2d_4485_9501_c3f7f06146d4.slice/crio-0d0775d066bb6f68f18c940467d1c02982e601fcb73df3dc77aacd0b596799b3 WatchSource:0}: Error finding container 0d0775d066bb6f68f18c940467d1c02982e601fcb73df3dc77aacd0b596799b3: Status 404 returned error can't find the container with id 0d0775d066bb6f68f18c940467d1c02982e601fcb73df3dc77aacd0b596799b3
Oct 31 00:44:52.664121 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.664099    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"]
Oct 31 00:44:52.665790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.665759    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:44:52.665899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.665821    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:44:52.665899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:52.665850    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:44:53.584902 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:53.584867    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" event={"ID":"98f5e737-fb2d-4485-9501-c3f7f06146d4","Type":"ContainerStarted","Data":"8b1f2d5b37e66e4c386bf799dfbdc7ffb79c35959bd3295d90e8ddec424b5208"}
Oct 31 00:44:53.585300 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:53.584926    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" event={"ID":"98f5e737-fb2d-4485-9501-c3f7f06146d4","Type":"ContainerStarted","Data":"0d0775d066bb6f68f18c940467d1c02982e601fcb73df3dc77aacd0b596799b3"}
Oct 31 00:44:53.643775 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:53.643725    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podStartSLOduration=1.643709753 podStartE2EDuration="1.643709753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:53.640760729 +0000 UTC m=+3061.587131447" watchObservedRunningTime="2026-10-31 00:44:53.643709753 +0000 UTC m=+3061.590080521"
Oct 31 00:44:54.380996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:54.380965    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:54.386002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:54.385970    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:54.590088 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:54.590061    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:44:54.590914 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:44:54.590896    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:45:00.200293 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.200261    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"]
Oct 31 00:45:00.204513 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.204489    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.215276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.215252    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:45:00.215414 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.215277    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:45:00.217515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.217494    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:45:00.232921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.232897    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"]
Oct 31 00:45:00.323930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.323899    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-8k47v\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.324109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.323977    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wxs\" (UniqueName: \"kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs\") pod \"maas-api-key-cleanup-29890125-8k47v\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.425006 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.424978    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-8k47v\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.425179 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.425052    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wxs\" (UniqueName: \"kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs\") pod \"maas-api-key-cleanup-29890125-8k47v\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.425632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.425611    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-8k47v\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.438476 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.438443    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wxs\" (UniqueName: \"kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs\") pod \"maas-api-key-cleanup-29890125-8k47v\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.518447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.518389    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:00.793119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:00.793094    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"]
Oct 31 00:45:00.794813 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:45:00.794788    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44d3756_d95d_4a61_86fe_c1f8491a5648.slice/crio-f0bf9bbc12ad572a1650355cc6a3803cfb911eb2c324b3d94ea9ab9636d0cf60 WatchSource:0}: Error finding container f0bf9bbc12ad572a1650355cc6a3803cfb911eb2c324b3d94ea9ab9636d0cf60: Status 404 returned error can't find the container with id f0bf9bbc12ad572a1650355cc6a3803cfb911eb2c324b3d94ea9ab9636d0cf60
Oct 31 00:45:01.628718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:01.628637    5078 generic.go:356] "Generic (PLEG): container finished" podID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerID="a6ed0f9b45a1a5b7584ce6a9dafbf4e94f6f05903481f7fc04e4b6c0760d3a4f" exitCode=60
Oct 31 00:45:01.628718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:01.628705    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" event={"ID":"a44d3756-d95d-4a61-86fe-c1f8491a5648","Type":"ContainerDied","Data":"a6ed0f9b45a1a5b7584ce6a9dafbf4e94f6f05903481f7fc04e4b6c0760d3a4f"}
Oct 31 00:45:01.629093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:01.628756    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" event={"ID":"a44d3756-d95d-4a61-86fe-c1f8491a5648","Type":"ContainerStarted","Data":"f0bf9bbc12ad572a1650355cc6a3803cfb911eb2c324b3d94ea9ab9636d0cf60"}
Oct 31 00:45:01.629093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:01.628932    5078 scope.go:122] "RemoveContainer" containerID="a6ed0f9b45a1a5b7584ce6a9dafbf4e94f6f05903481f7fc04e4b6c0760d3a4f"
Oct 31 00:45:02.635581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:02.635551    5078 generic.go:356] "Generic (PLEG): container finished" podID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerID="6f59977465f737227150f5df573e5088d3636dda47da5fc024fa106a3ad543b9" exitCode=60
Oct 31 00:45:02.636060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:02.635595    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" event={"ID":"a44d3756-d95d-4a61-86fe-c1f8491a5648","Type":"ContainerDied","Data":"6f59977465f737227150f5df573e5088d3636dda47da5fc024fa106a3ad543b9"}
Oct 31 00:45:02.636060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:02.635658    5078 scope.go:122] "RemoveContainer" containerID="a6ed0f9b45a1a5b7584ce6a9dafbf4e94f6f05903481f7fc04e4b6c0760d3a4f"
Oct 31 00:45:02.636060 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:02.635938    5078 scope.go:122] "RemoveContainer" containerID="6f59977465f737227150f5df573e5088d3636dda47da5fc024fa106a3ad543b9"
Oct 31 00:45:02.636259 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:45:02.636235    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-8k47v_odh-ai-gateway-infra(a44d3756-d95d-4a61-86fe-c1f8491a5648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648"
Oct 31 00:45:03.644113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:03.644087    5078 scope.go:122] "RemoveContainer" containerID="6f59977465f737227150f5df573e5088d3636dda47da5fc024fa106a3ad543b9"
Oct 31 00:45:03.644554 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:45:03.644329    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-8k47v_odh-ai-gateway-infra(a44d3756-d95d-4a61-86fe-c1f8491a5648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648"
Oct 31 00:45:06.386869 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:45:06.386830    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:45:07.078177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.078146    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"]
Oct 31 00:45:07.083901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.083879    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.088698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.088676    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.088808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.088709    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fbc0c4-9520-446a-826b-20866760c19a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.088808 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.088736    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/e1fbc0c4-9520-446a-826b-20866760c19a-kube-api-access-thzbv\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.088891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.088847    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.088891 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.088871    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.088972 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.088956    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fbc0c4-9520-446a-826b-20866760c19a-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.089012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.089003    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.089047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.089021    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fbc0c4-9520-446a-826b-20866760c19a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.089097 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.089054    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.096988 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.096963    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"]
Oct 31 00:45:07.189572 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.189546    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fbc0c4-9520-446a-826b-20866760c19a-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.189749 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.189710    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.189838 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.189810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fbc0c4-9520-446a-826b-20866760c19a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.189932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.189911    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.189989    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190042    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fbc0c4-9520-446a-826b-20866760c19a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190084    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/e1fbc0c4-9520-446a-826b-20866760c19a-kube-api-access-thzbv\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190159 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190123    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190177    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190237    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190352 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190311    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190366    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190553    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.190806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.190749    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fbc0c4-9520-446a-826b-20866760c19a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.192439 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.192419    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fbc0c4-9520-446a-826b-20866760c19a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.192601 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.192585    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fbc0c4-9520-446a-826b-20866760c19a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.203018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.202993    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fbc0c4-9520-446a-826b-20866760c19a-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.203372 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.203354    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/e1fbc0c4-9520-446a-826b-20866760c19a-kube-api-access-thzbv\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-7kqrm\" (UID: \"e1fbc0c4-9520-446a-826b-20866760c19a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.401242 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.401213    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:07.689116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.689089    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"]
Oct 31 00:45:07.690038 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:45:07.690001    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fbc0c4_9520_446a_826b_20866760c19a.slice/crio-b244863bd09ebcb9c63e71314e4a1664744b5bcef574ea9204e554e60e3472c5 WatchSource:0}: Error finding container b244863bd09ebcb9c63e71314e4a1664744b5bcef574ea9204e554e60e3472c5: Status 404 returned error can't find the container with id b244863bd09ebcb9c63e71314e4a1664744b5bcef574ea9204e554e60e3472c5
Oct 31 00:45:07.692187 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.692158    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:45:07.692278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.692217    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:45:07.692278 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:07.692245    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:45:08.671238 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:08.671187    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" event={"ID":"e1fbc0c4-9520-446a-826b-20866760c19a","Type":"ContainerStarted","Data":"560f6e48feaddf83bec992e1fa4a081371d0b0eae23a6e24ea1aec7f60ae9d93"}
Oct 31 00:45:08.671754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:08.671248    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" event={"ID":"e1fbc0c4-9520-446a-826b-20866760c19a","Type":"ContainerStarted","Data":"b244863bd09ebcb9c63e71314e4a1664744b5bcef574ea9204e554e60e3472c5"}
Oct 31 00:45:08.701391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:08.701349    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podStartSLOduration=1.70133621 podStartE2EDuration="1.70133621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:08.698387415 +0000 UTC m=+3076.644758129" watchObservedRunningTime="2026-10-31 00:45:08.70133621 +0000 UTC m=+3076.647706977"
Oct 31 00:45:09.401657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:09.401618    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:09.406443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:09.406422    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:09.676744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:09.676669    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:09.677522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:09.677505    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:45:16.187058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:16.187027    5078 scope.go:122] "RemoveContainer" containerID="6f59977465f737227150f5df573e5088d3636dda47da5fc024fa106a3ad543b9"
Oct 31 00:45:16.714843 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:16.714808    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" event={"ID":"a44d3756-d95d-4a61-86fe-c1f8491a5648","Type":"ContainerStarted","Data":"19042a826ce500aeec637547050588314f848d0b2ff10bfed30159122d6aa356"}
Oct 31 00:45:16.735618 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:16.735519    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" podStartSLOduration=16.735505709999998 podStartE2EDuration="16.73550571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:16.734796775 +0000 UTC m=+3084.681167493" watchObservedRunningTime="2026-10-31 00:45:16.73550571 +0000 UTC m=+3084.681876426"
Oct 31 00:45:17.722860 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:17.722822    5078 generic.go:356] "Generic (PLEG): container finished" podID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerID="19042a826ce500aeec637547050588314f848d0b2ff10bfed30159122d6aa356" exitCode=60
Oct 31 00:45:17.723320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:17.722892    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" event={"ID":"a44d3756-d95d-4a61-86fe-c1f8491a5648","Type":"ContainerDied","Data":"19042a826ce500aeec637547050588314f848d0b2ff10bfed30159122d6aa356"}
Oct 31 00:45:17.723320 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:17.722979    5078 scope.go:122] "RemoveContainer" containerID="6f59977465f737227150f5df573e5088d3636dda47da5fc024fa106a3ad543b9"
Oct 31 00:45:17.723487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:17.723322    5078 scope.go:122] "RemoveContainer" containerID="19042a826ce500aeec637547050588314f848d0b2ff10bfed30159122d6aa356"
Oct 31 00:45:17.723645 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:45:17.723621    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-8k47v_odh-ai-gateway-infra(a44d3756-d95d-4a61-86fe-c1f8491a5648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648"
Oct 31 00:45:17.756268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:17.756241    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"]
Oct 31 00:45:18.875284 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:18.875259    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:19.012099 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.012012    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca\") pod \"a44d3756-d95d-4a61-86fe-c1f8491a5648\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") "
Oct 31 00:45:19.012271 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.012137    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs\" (UniqueName: \"kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs\") pod \"a44d3756-d95d-4a61-86fe-c1f8491a5648\" (UID: \"a44d3756-d95d-4a61-86fe-c1f8491a5648\") "
Oct 31 00:45:19.012364 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.012342    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca" pod "a44d3756-d95d-4a61-86fe-c1f8491a5648" (UID: "a44d3756-d95d-4a61-86fe-c1f8491a5648"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:19.012430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.012420    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44d3756-d95d-4a61-86fe-c1f8491a5648-maas-api-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:19.014438 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.014415    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs" pod "a44d3756-d95d-4a61-86fe-c1f8491a5648" (UID: "a44d3756-d95d-4a61-86fe-c1f8491a5648"). InnerVolumeSpecName "kube-api-access-95wxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:19.113107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.113082    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wxs\" (UniqueName: \"kubernetes.io/projected/a44d3756-d95d-4a61-86fe-c1f8491a5648-kube-api-access-95wxs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:19.736579 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.736551    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"
Oct 31 00:45:19.736778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.736586    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v" event={"ID":"a44d3756-d95d-4a61-86fe-c1f8491a5648","Type":"ContainerDied","Data":"f0bf9bbc12ad572a1650355cc6a3803cfb911eb2c324b3d94ea9ab9636d0cf60"}
Oct 31 00:45:19.736778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.736645    5078 scope.go:122] "RemoveContainer" containerID="19042a826ce500aeec637547050588314f848d0b2ff10bfed30159122d6aa356"
Oct 31 00:45:19.763527 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.763505    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"]
Oct 31 00:45:19.766344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:19.766325    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-8k47v"]
Oct 31 00:45:20.193479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:20.193437    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" path="/var/lib/kubelet/pods/a44d3756-d95d-4a61-86fe-c1f8491a5648/volumes"
Oct 31 00:45:36.386030 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:45:36.385981    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:45:50.177113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177079    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"]
Oct 31 00:45:50.177576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177562    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:45:50.177651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177577    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:45:50.177651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177598    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:45:50.177651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177603    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:45:50.177751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177685    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:45:50.177751 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.177698    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:45:50.182339 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.182323    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.192767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.192693    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-frf6h\""
Oct 31 00:45:50.195126 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.195104    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"]
Oct 31 00:45:50.203553 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.203531    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0173b8-debd-4739-80ee-070c7445a663-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cv2kn\" (UID: \"eb0173b8-debd-4739-80ee-070c7445a663\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.203658 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.203568    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zpb\" (UniqueName: \"kubernetes.io/projected/eb0173b8-debd-4739-80ee-070c7445a663-kube-api-access-64zpb\") pod \"kuadrant-operator-controller-manager-678d87766f-cv2kn\" (UID: \"eb0173b8-debd-4739-80ee-070c7445a663\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.305949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.305177    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0173b8-debd-4739-80ee-070c7445a663-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cv2kn\" (UID: \"eb0173b8-debd-4739-80ee-070c7445a663\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.305949 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.305235    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zpb\" (UniqueName: \"kubernetes.io/projected/eb0173b8-debd-4739-80ee-070c7445a663-kube-api-access-64zpb\") pod \"kuadrant-operator-controller-manager-678d87766f-cv2kn\" (UID: \"eb0173b8-debd-4739-80ee-070c7445a663\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.306486 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.306362    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0173b8-debd-4739-80ee-070c7445a663-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cv2kn\" (UID: \"eb0173b8-debd-4739-80ee-070c7445a663\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.319801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.319771    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zpb\" (UniqueName: \"kubernetes.io/projected/eb0173b8-debd-4739-80ee-070c7445a663-kube-api-access-64zpb\") pod \"kuadrant-operator-controller-manager-678d87766f-cv2kn\" (UID: \"eb0173b8-debd-4739-80ee-070c7445a663\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.497991 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.497908    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.793133 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.793107    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"]
Oct 31 00:45:50.795066 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:45:50.795039    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0173b8_debd_4739_80ee_070c7445a663.slice/crio-a670994dd227d99f8fa397b5d992406c694079274386b92a8ce9f5694f15a53e WatchSource:0}: Error finding container a670994dd227d99f8fa397b5d992406c694079274386b92a8ce9f5694f15a53e: Status 404 returned error can't find the container with id a670994dd227d99f8fa397b5d992406c694079274386b92a8ce9f5694f15a53e
Oct 31 00:45:50.899681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.899650    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn" event={"ID":"eb0173b8-debd-4739-80ee-070c7445a663","Type":"ContainerStarted","Data":"5bbbfeb283d9fcdf5b362c891c461fbe9a7178248f83ba1184d3d1df9c26760c"}
Oct 31 00:45:50.899884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.899714    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn" event={"ID":"eb0173b8-debd-4739-80ee-070c7445a663","Type":"ContainerStarted","Data":"a670994dd227d99f8fa397b5d992406c694079274386b92a8ce9f5694f15a53e"}
Oct 31 00:45:50.899884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.899845    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:45:50.928315 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:45:50.928267    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn" podStartSLOduration=0.928253435 podStartE2EDuration="928.253435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:50.926006134 +0000 UTC m=+3118.872376851" watchObservedRunningTime="2026-10-31 00:45:50.928253435 +0000 UTC m=+3118.874624148"
Oct 31 00:46:01.907371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:01.907339    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cv2kn"
Oct 31 00:46:06.388749 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:46:06.388709    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:46:15.236825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.236789    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"]
Oct 31 00:46:15.237326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.237309    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:46:15.237326 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.237324    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:46:15.237442 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.237426    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44d3756-d95d-4a61-86fe-c1f8491a5648" containerName="cleanup"
Oct 31 00:46:15.241790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.241770    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.248515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.248431    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:46:15.248515 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.248438    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:46:15.248726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.248550    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:46:15.248726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.248689    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:46:15.248962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.248943    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"]
Oct 31 00:46:15.343683 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.343651    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.343896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.343741    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzmb\" (UniqueName: \"kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.343896 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.343879    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.445396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.445357    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.445597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.445521    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzmb\" (UniqueName: \"kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.445597 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.445589    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.446301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.446280    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.447940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.447923    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.459210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.459190    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzmb\" (UniqueName: \"kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb\") pod \"maas-api-revoke-sub-d576e68b-psp8j\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.558377 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.558286    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:15.905883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:15.896444    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"]
Oct 31 00:46:15.911228 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:46:15.911192    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551413cd_902f_4a94_a697_0e4ed15b968f.slice/crio-0ef94daf7cd14d6da121484cbfec6fcd5e48a9963cc7fcf4d07dd46ec85380fe WatchSource:0}: Error finding container 0ef94daf7cd14d6da121484cbfec6fcd5e48a9963cc7fcf4d07dd46ec85380fe: Status 404 returned error can't find the container with id 0ef94daf7cd14d6da121484cbfec6fcd5e48a9963cc7fcf4d07dd46ec85380fe
Oct 31 00:46:16.118274 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:16.118240    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j" event={"ID":"551413cd-902f-4a94-a697-0e4ed15b968f","Type":"ContainerStarted","Data":"0ef94daf7cd14d6da121484cbfec6fcd5e48a9963cc7fcf4d07dd46ec85380fe"}
Oct 31 00:46:17.130940 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:17.130848    5078 generic.go:356] "Generic (PLEG): container finished" podID="551413cd-902f-4a94-a697-0e4ed15b968f" containerID="90718bc14604cce6f0105b56af446b18ef24b85093a3917e6755db7d26909361" exitCode=0
Oct 31 00:46:17.131415 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:17.130925    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j" event={"ID":"551413cd-902f-4a94-a697-0e4ed15b968f","Type":"ContainerDied","Data":"90718bc14604cce6f0105b56af446b18ef24b85093a3917e6755db7d26909361"}
Oct 31 00:46:18.343805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.343776    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:18.377265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.377215    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca\") pod \"551413cd-902f-4a94-a697-0e4ed15b968f\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") "
Oct 31 00:46:18.377265 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.377270    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca\") pod \"551413cd-902f-4a94-a697-0e4ed15b968f\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") "
Oct 31 00:46:18.377554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.377348    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb\" (UniqueName: \"kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb\") pod \"551413cd-902f-4a94-a697-0e4ed15b968f\" (UID: \"551413cd-902f-4a94-a697-0e4ed15b968f\") "
Oct 31 00:46:18.378072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.377964    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca" pod "551413cd-902f-4a94-a697-0e4ed15b968f" (UID: "551413cd-902f-4a94-a697-0e4ed15b968f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:18.380430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.380404    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca" pod "551413cd-902f-4a94-a697-0e4ed15b968f" (UID: "551413cd-902f-4a94-a697-0e4ed15b968f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:18.385694 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.385666    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb" pod "551413cd-902f-4a94-a697-0e4ed15b968f" (UID: "551413cd-902f-4a94-a697-0e4ed15b968f"). InnerVolumeSpecName "kube-api-access-6kzmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:18.478424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.478384    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:18.478612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.478441    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551413cd-902f-4a94-a697-0e4ed15b968f-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:18.478612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:18.478453    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzmb\" (UniqueName: \"kubernetes.io/projected/551413cd-902f-4a94-a697-0e4ed15b968f-kube-api-access-6kzmb\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:19.144021 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:19.143984    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"
Oct 31 00:46:19.144196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:19.144007    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j" event={"ID":"551413cd-902f-4a94-a697-0e4ed15b968f","Type":"ContainerDied","Data":"0ef94daf7cd14d6da121484cbfec6fcd5e48a9963cc7fcf4d07dd46ec85380fe"}
Oct 31 00:46:19.144196 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:19.144098    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef94daf7cd14d6da121484cbfec6fcd5e48a9963cc7fcf4d07dd46ec85380fe"
Oct 31 00:46:21.156101 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:21.156073    5078 generic.go:356] "Generic (PLEG): container finished" podID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerID="8b1f2d5b37e66e4c386bf799dfbdc7ffb79c35959bd3295d90e8ddec424b5208" exitCode=0
Oct 31 00:46:21.156508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:21.156115    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" event={"ID":"98f5e737-fb2d-4485-9501-c3f7f06146d4","Type":"ContainerDied","Data":"8b1f2d5b37e66e4c386bf799dfbdc7ffb79c35959bd3295d90e8ddec424b5208"}
Oct 31 00:46:21.156508 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:21.156497    5078 scope.go:122] "RemoveContainer" containerID="8b1f2d5b37e66e4c386bf799dfbdc7ffb79c35959bd3295d90e8ddec424b5208"
Oct 31 00:46:21.158613 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:21.158582    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:46:21.158716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:21.158670    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:46:21.158716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:21.158704    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:46:22.163012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.162975    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" event={"ID":"98f5e737-fb2d-4485-9501-c3f7f06146d4","Type":"ContainerStarted","Data":"c2bfe7110b2d086851d89c54ed65c23060d7b557021b256efdce6002fba8aafa"}
Oct 31 00:46:22.164576 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.164553    5078 generic.go:356] "Generic (PLEG): container finished" podID="e1fbc0c4-9520-446a-826b-20866760c19a" containerID="560f6e48feaddf83bec992e1fa4a081371d0b0eae23a6e24ea1aec7f60ae9d93" exitCode=0
Oct 31 00:46:22.164701 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.164632    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" event={"ID":"e1fbc0c4-9520-446a-826b-20866760c19a","Type":"ContainerDied","Data":"560f6e48feaddf83bec992e1fa4a081371d0b0eae23a6e24ea1aec7f60ae9d93"}
Oct 31 00:46:22.164981 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.164966    5078 scope.go:122] "RemoveContainer" containerID="560f6e48feaddf83bec992e1fa4a081371d0b0eae23a6e24ea1aec7f60ae9d93"
Oct 31 00:46:22.166109 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.166081    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:46:22.166197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.166145    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:46:22.166197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.166181    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:46:22.389884 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.389851    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:46:22.394697 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.394661    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:22.394915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:22.394741    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:23.214487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:23.214180    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" event={"ID":"e1fbc0c4-9520-446a-826b-20866760c19a","Type":"ContainerStarted","Data":"4a1f45e10e614439073783f319f87d105fc96059c70a3ca2e72e808d259e2a3d"}
Oct 31 00:46:23.383026 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:23.382990    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:23.383309 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:23.383280    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:23.408490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:23.406815    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:46:23.412912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:23.412577    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:23.412912 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:23.412636    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:24.393158 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:24.391802    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:24.393834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:24.393802    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:24.417483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:24.410798    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:24.417483 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:24.410840    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:24.594491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:24.594232    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:46:24.686545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:24.677551    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:46:25.384215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:25.383790    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:25.384215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:25.383852    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:25.404920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:25.404815    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:25.404920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:25.404870    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:26.382899 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:26.382862    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:26.383200 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:26.383174    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:26.404855 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:26.404822    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:26.405170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:26.405139    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:27.391488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:27.390602    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:27.391488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:27.390668    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:27.402479 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:27.402431    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:27.402633 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:27.402505    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:28.380721 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:28.380668    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 00:46:28.381259 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:28.380743    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 00:46:28.401647 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:28.401605    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:28.401834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:28.401656    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:29.401757 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:29.401717    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Oct 31 00:46:29.402322 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:29.401784    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Oct 31 00:46:30.386518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:30.382560    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:30.386518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:30.382627    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:31.408640 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:31.408591    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:31.409336 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:31.409292    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:31.448487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:31.446577    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:31.448487 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:31.446655    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:32.450656 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:32.450611    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:32.451113 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:32.450680    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:32.488488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:32.487652    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:32.488488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:32.487786    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:33.488889 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:33.488828    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:33.489447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:33.488940    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:33.491660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:33.491610    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:33.491825 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:33.491663    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:34.492781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:34.492332    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:34.492781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:34.492410    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:34.531668 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:34.531630    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:34.531848 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:34.531697    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:35.310779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:35.310739    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:46:35.313632 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:35.313575    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj"
Oct 31 00:46:35.533641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:35.533600    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:46:35.534171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:35.533670    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm" podUID="e1fbc0c4-9520-446a-826b-20866760c19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:46:36.186163 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.186129    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:46:36.193083 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.193061    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-7kqrm"
Oct 31 00:46:36.394180 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:46:36.394137    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:46:36.602602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.602513    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"]
Oct 31 00:46:36.603107 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.603090    5078 cpu_manager.go:414] "containerMap: removing container" podUID="551413cd-902f-4a94-a697-0e4ed15b968f" containerName="revoke-keys"
Oct 31 00:46:36.603149 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.603109    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551413cd-902f-4a94-a697-0e4ed15b968f" containerName="revoke-keys"
Oct 31 00:46:36.603204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.603195    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="551413cd-902f-4a94-a697-0e4ed15b968f" containerName="revoke-keys"
Oct 31 00:46:36.606547 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.606531    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.616192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.615811    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:46:36.616287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.616238    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:46:36.617177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.616510    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:46:36.617177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.616955    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:46:36.627675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.627641    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"]
Oct 31 00:46:36.758673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.758636    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.758866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.758745    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.758866 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.758823    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfcw\" (UniqueName: \"kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.860413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.860336    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfcw\" (UniqueName: \"kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.860413 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.860402    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.860660 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.860445    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.861091 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.861064    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.863093 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.863072    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.874552 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.874529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfcw\" (UniqueName: \"kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw\") pod \"maas-api-revoke-sub-d2bc93bc-rqcsn\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:36.936993 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:36.936963    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:37.213033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:37.213004    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"]
Oct 31 00:46:37.214566 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:46:37.214537    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6072c2_5488_43fd_b49f_52516d75e4f6.slice/crio-017a9f4baf741300e66974f65afac8c1d4da1bf4616e8fdf6fe3a1303419fab3 WatchSource:0}: Error finding container 017a9f4baf741300e66974f65afac8c1d4da1bf4616e8fdf6fe3a1303419fab3: Status 404 returned error can't find the container with id 017a9f4baf741300e66974f65afac8c1d4da1bf4616e8fdf6fe3a1303419fab3
Oct 31 00:46:37.331713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:37.331681    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn" event={"ID":"6f6072c2-5488-43fd-b49f-52516d75e4f6","Type":"ContainerStarted","Data":"017a9f4baf741300e66974f65afac8c1d4da1bf4616e8fdf6fe3a1303419fab3"}
Oct 31 00:46:38.338699 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:38.338607    5078 generic.go:356] "Generic (PLEG): container finished" podID="6f6072c2-5488-43fd-b49f-52516d75e4f6" containerID="3ce92bf98407cc232bac516500862154fd3e22193ae936232a0dc0a1257d09a2" exitCode=0
Oct 31 00:46:38.339086 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:38.338688    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn" event={"ID":"6f6072c2-5488-43fd-b49f-52516d75e4f6","Type":"ContainerDied","Data":"3ce92bf98407cc232bac516500862154fd3e22193ae936232a0dc0a1257d09a2"}
Oct 31 00:46:39.488943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.488919    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:39.588344 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.588309    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca\") pod \"6f6072c2-5488-43fd-b49f-52516d75e4f6\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") "
Oct 31 00:46:39.588558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.588368    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw\" (UniqueName: \"kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw\") pod \"6f6072c2-5488-43fd-b49f-52516d75e4f6\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") "
Oct 31 00:46:39.588558 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.588518    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca\") pod \"6f6072c2-5488-43fd-b49f-52516d75e4f6\" (UID: \"6f6072c2-5488-43fd-b49f-52516d75e4f6\") "
Oct 31 00:46:39.588881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.588860    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca" pod "6f6072c2-5488-43fd-b49f-52516d75e4f6" (UID: "6f6072c2-5488-43fd-b49f-52516d75e4f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:39.590518 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.590500    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca" pod "6f6072c2-5488-43fd-b49f-52516d75e4f6" (UID: "6f6072c2-5488-43fd-b49f-52516d75e4f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:39.590592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.590570    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw" pod "6f6072c2-5488-43fd-b49f-52516d75e4f6" (UID: "6f6072c2-5488-43fd-b49f-52516d75e4f6"). InnerVolumeSpecName "kube-api-access-zhfcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:39.689973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.689946    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:39.689973 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.689969    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfcw\" (UniqueName: \"kubernetes.io/projected/6f6072c2-5488-43fd-b49f-52516d75e4f6-kube-api-access-zhfcw\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:39.690177 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:39.689981    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6072c2-5488-43fd-b49f-52516d75e4f6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:40.349580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:40.349549    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"
Oct 31 00:46:40.349746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:40.349586    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn" event={"ID":"6f6072c2-5488-43fd-b49f-52516d75e4f6","Type":"ContainerDied","Data":"017a9f4baf741300e66974f65afac8c1d4da1bf4616e8fdf6fe3a1303419fab3"}
Oct 31 00:46:40.349746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:40.349631    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="017a9f4baf741300e66974f65afac8c1d4da1bf4616e8fdf6fe3a1303419fab3"
Oct 31 00:46:53.470706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.470672    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"]
Oct 31 00:46:53.471170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.471154    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6f6072c2-5488-43fd-b49f-52516d75e4f6" containerName="revoke-keys"
Oct 31 00:46:53.471249 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.471171    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6072c2-5488-43fd-b49f-52516d75e4f6" containerName="revoke-keys"
Oct 31 00:46:53.471308 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.471269    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6072c2-5488-43fd-b49f-52516d75e4f6" containerName="revoke-keys"
Oct 31 00:46:53.474529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.474513    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.481024 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.480998    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"]
Oct 31 00:46:53.484806 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.484778    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"]
Oct 31 00:46:53.484987 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.484966    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.486921 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.486904    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b0db87\""
Oct 31 00:46:53.487056 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.487003    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-97fd441c-dockercfg-bzwzz\""
Oct 31 00:46:53.495673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.495652    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"]
Oct 31 00:46:53.522270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.522103    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.522270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.522150    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4sck\" (UniqueName: \"kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck\") pod \"payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.522270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.522257    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hh2\" (UniqueName: \"kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2\") pod \"payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.522516 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.522383    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.623450 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.623413    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4sck\" (UniqueName: \"kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck\") pod \"payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.623654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.623526    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hh2\" (UniqueName: \"kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2\") pod \"payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.623654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.623562    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.623654 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.623626    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.624592 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.624570    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.624661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.624580    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.637178 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.637155    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hh2\" (UniqueName: \"kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2\") pod \"payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.637382 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.637357    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4sck\" (UniqueName: \"kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck\") pod \"payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:53.792012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.791918    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:53.805698 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:53.805673    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:54.088675 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.088651    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"]
Oct 31 00:46:54.090633 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:46:54.090602    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a29cf60_52ea_4968_9a8b_068513cc3553.slice/crio-f8c38af40978be77087945f46a3e42a94c8b543ee0e8fb19b97552eed5845cdd WatchSource:0}: Error finding container f8c38af40978be77087945f46a3e42a94c8b543ee0e8fb19b97552eed5845cdd: Status 404 returned error can't find the container with id f8c38af40978be77087945f46a3e42a94c8b543ee0e8fb19b97552eed5845cdd
Oct 31 00:46:54.100401 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.100376    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"]
Oct 31 00:46:54.103436 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:46:54.103413    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b01f61b_3974_4fe0_9f3e_db7af5dc4401.slice/crio-2528b817f228fef741caaf465e3f4a9eb81d2af7d0b864d8c35fd7b48275ccd0 WatchSource:0}: Error finding container 2528b817f228fef741caaf465e3f4a9eb81d2af7d0b864d8c35fd7b48275ccd0: Status 404 returned error can't find the container with id 2528b817f228fef741caaf465e3f4a9eb81d2af7d0b864d8c35fd7b48275ccd0
Oct 31 00:46:54.427624 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.427585    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" event={"ID":"7a29cf60-52ea-4968-9a8b-068513cc3553","Type":"ContainerStarted","Data":"b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6"}
Oct 31 00:46:54.427828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.427660    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:46:54.427828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.427687    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" event={"ID":"7a29cf60-52ea-4968-9a8b-068513cc3553","Type":"ContainerStarted","Data":"f8c38af40978be77087945f46a3e42a94c8b543ee0e8fb19b97552eed5845cdd"}
Oct 31 00:46:54.429330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.429301    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" event={"ID":"1b01f61b-3974-4fe0-9f3e-db7af5dc4401","Type":"ContainerStarted","Data":"7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7"}
Oct 31 00:46:54.429452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.429360    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:46:54.429452 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.429381    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" event={"ID":"1b01f61b-3974-4fe0-9f3e-db7af5dc4401","Type":"ContainerStarted","Data":"2528b817f228fef741caaf465e3f4a9eb81d2af7d0b864d8c35fd7b48275ccd0"}
Oct 31 00:46:54.480712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.480661    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" podStartSLOduration=1.480643202 podStartE2EDuration="1.480643202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:54.479203102 +0000 UTC m=+3182.425573820" watchObservedRunningTime="2026-10-31 00:46:54.480643202 +0000 UTC m=+3182.427013919"
Oct 31 00:46:54.515201 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:46:54.515151    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" podStartSLOduration=1.515139553 podStartE2EDuration="1.515139553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:54.513640597 +0000 UTC m=+3182.460011315" watchObservedRunningTime="2026-10-31 00:46:54.515139553 +0000 UTC m=+3182.461510270"
Oct 31 00:47:05.438602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:05.438527    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:47:05.438602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:05.438601    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:47:06.385782 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:47:06.385746    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:47:36.386347 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:47:36.386303    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:47:39.816129 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.816098    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"]
Oct 31 00:47:39.820141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.820126    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:39.830950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.830927    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:47:39.832674 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.832650    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:47:39.832802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.832723    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:47:39.832802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.832795    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:47:39.836447 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.836426    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"]
Oct 31 00:47:39.910986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.910939    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:39.910986 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.910992    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwf8r\" (UniqueName: \"kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:39.911323 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:39.911220    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.012118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.012079    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.012118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.012122    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwf8r\" (UniqueName: \"kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.012354 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.012185    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.012887 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.012864    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.014805 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.014782    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.025784 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.025757    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwf8r\" (UniqueName: \"kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r\") pod \"maas-api-revoke-sub-c6289aae-nnb8l\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.134314 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.134288    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:40.420069 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.420041    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"]
Oct 31 00:47:40.421051 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:47:40.421027    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f1979f_da2d_42f2_a47e_b901809ab96e.slice/crio-e84ac99141f8173e750647852c448307b4b64525d6cd8ce0c33ad81c3b82a297 WatchSource:0}: Error finding container e84ac99141f8173e750647852c448307b4b64525d6cd8ce0c33ad81c3b82a297: Status 404 returned error can't find the container with id e84ac99141f8173e750647852c448307b4b64525d6cd8ce0c33ad81c3b82a297
Oct 31 00:47:40.699681 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.699649    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l" event={"ID":"52f1979f-da2d-42f2-a47e-b901809ab96e","Type":"ContainerStarted","Data":"a9db68b4047b2198e553ed146693853a6470cf407f03dd6b7600538194466061"}
Oct 31 00:47:40.699844 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.699705    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l" event={"ID":"52f1979f-da2d-42f2-a47e-b901809ab96e","Type":"ContainerStarted","Data":"e84ac99141f8173e750647852c448307b4b64525d6cd8ce0c33ad81c3b82a297"}
Oct 31 00:47:40.726275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:40.726225    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l" podStartSLOduration=1.726211833 podStartE2EDuration="1.726211833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:40.72397826 +0000 UTC m=+3228.670348973" watchObservedRunningTime="2026-10-31 00:47:40.726211833 +0000 UTC m=+3228.672582550"
Oct 31 00:47:41.707253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:41.707219    5078 generic.go:356] "Generic (PLEG): container finished" podID="52f1979f-da2d-42f2-a47e-b901809ab96e" containerID="a9db68b4047b2198e553ed146693853a6470cf407f03dd6b7600538194466061" exitCode=0
Oct 31 00:47:41.707778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:41.707301    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l" event={"ID":"52f1979f-da2d-42f2-a47e-b901809ab96e","Type":"ContainerDied","Data":"a9db68b4047b2198e553ed146693853a6470cf407f03dd6b7600538194466061"}
Oct 31 00:47:42.866378 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.866356    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:42.943585 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.943559    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca\") pod \"52f1979f-da2d-42f2-a47e-b901809ab96e\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") "
Oct 31 00:47:42.943744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.943681    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r\" (UniqueName: \"kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r\") pod \"52f1979f-da2d-42f2-a47e-b901809ab96e\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") "
Oct 31 00:47:42.943744 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.943741    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca\") pod \"52f1979f-da2d-42f2-a47e-b901809ab96e\" (UID: \"52f1979f-da2d-42f2-a47e-b901809ab96e\") "
Oct 31 00:47:42.944116 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.944090    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca" pod "52f1979f-da2d-42f2-a47e-b901809ab96e" (UID: "52f1979f-da2d-42f2-a47e-b901809ab96e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:47:42.945802 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.945784    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca" pod "52f1979f-da2d-42f2-a47e-b901809ab96e" (UID: "52f1979f-da2d-42f2-a47e-b901809ab96e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:47:42.945957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:42.945942    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r" pod "52f1979f-da2d-42f2-a47e-b901809ab96e" (UID: "52f1979f-da2d-42f2-a47e-b901809ab96e"). InnerVolumeSpecName "kube-api-access-dwf8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:47:43.044673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:43.044614    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwf8r\" (UniqueName: \"kubernetes.io/projected/52f1979f-da2d-42f2-a47e-b901809ab96e-kube-api-access-dwf8r\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:43.044673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:43.044637    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:43.044673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:43.044648    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f1979f-da2d-42f2-a47e-b901809ab96e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:43.722270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:43.722235    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"
Oct 31 00:47:43.722270 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:43.722261    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l" event={"ID":"52f1979f-da2d-42f2-a47e-b901809ab96e","Type":"ContainerDied","Data":"e84ac99141f8173e750647852c448307b4b64525d6cd8ce0c33ad81c3b82a297"}
Oct 31 00:47:43.722530 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:43.722309    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84ac99141f8173e750647852c448307b4b64525d6cd8ce0c33ad81c3b82a297"
Oct 31 00:47:50.826015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.825965    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"]
Oct 31 00:47:50.826682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.826660    5078 cpu_manager.go:414] "containerMap: removing container" podUID="52f1979f-da2d-42f2-a47e-b901809ab96e" containerName="revoke-keys"
Oct 31 00:47:50.826788 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.826683    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f1979f-da2d-42f2-a47e-b901809ab96e" containerName="revoke-keys"
Oct 31 00:47:50.826897 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.826883    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f1979f-da2d-42f2-a47e-b901809ab96e" containerName="revoke-keys"
Oct 31 00:47:50.830422 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.830402    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:50.835155 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.835132    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:47:50.837224 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.837205    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:47:50.837370 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.837268    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:47:50.837584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.837559    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:47:50.841753 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.841728    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"]
Oct 31 00:47:50.920828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.920798    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:50.920996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.920871    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:50.921041 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:50.920985    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrwc\" (UniqueName: \"kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.022263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.022231    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.022449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.022288    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.022449 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.022311    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrwc\" (UniqueName: \"kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.023047 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.023020    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.024858 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.024835    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.035276 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.035255    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrwc\" (UniqueName: \"kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.146282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.146241    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:51.431960 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.431931    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"]
Oct 31 00:47:51.433892 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:47:51.433860    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-2de9ccab38e5479ee832749d83ded2cc53c34b7c0c58c0dbc32c650f4c00b0cc WatchSource:0}: Error finding container 2de9ccab38e5479ee832749d83ded2cc53c34b7c0c58c0dbc32c650f4c00b0cc: Status 404 returned error can't find the container with id 2de9ccab38e5479ee832749d83ded2cc53c34b7c0c58c0dbc32c650f4c00b0cc
Oct 31 00:47:51.772687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.772654    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm" event={"ID":"43ec38ca-388e-41d7-bdd9-2c98b3288abe","Type":"ContainerStarted","Data":"d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b"}
Oct 31 00:47:51.772864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.772720    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm" event={"ID":"43ec38ca-388e-41d7-bdd9-2c98b3288abe","Type":"ContainerStarted","Data":"2de9ccab38e5479ee832749d83ded2cc53c34b7c0c58c0dbc32c650f4c00b0cc"}
Oct 31 00:47:51.810431 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:51.810383    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm" podStartSLOduration=1.8103684979999999 podStartE2EDuration="1.810368498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:51.809561138 +0000 UTC m=+3239.755931854" watchObservedRunningTime="2026-10-31 00:47:51.810368498 +0000 UTC m=+3239.756739214"
Oct 31 00:47:52.781248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:52.781208    5078 generic.go:356] "Generic (PLEG): container finished" podID="43ec38ca-388e-41d7-bdd9-2c98b3288abe" containerID="d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b" exitCode=0
Oct 31 00:47:52.781700 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:52.781291    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm" event={"ID":"43ec38ca-388e-41d7-bdd9-2c98b3288abe","Type":"ContainerDied","Data":"d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b"}
Oct 31 00:47:53.939943 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:53.939907    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:53.982282 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:47:53.982248    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:47:54.053351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.053271    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca\") pod \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") "
Oct 31 00:47:54.053351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.053305    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc\" (UniqueName: \"kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc\") pod \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") "
Oct 31 00:47:54.053351 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.053333    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca\") pod \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\" (UID: \"43ec38ca-388e-41d7-bdd9-2c98b3288abe\") "
Oct 31 00:47:54.053749 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.053726    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca" pod "43ec38ca-388e-41d7-bdd9-2c98b3288abe" (UID: "43ec38ca-388e-41d7-bdd9-2c98b3288abe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:47:54.055693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.055672    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca" pod "43ec38ca-388e-41d7-bdd9-2c98b3288abe" (UID: "43ec38ca-388e-41d7-bdd9-2c98b3288abe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:47:54.055829 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.055814    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc" pod "43ec38ca-388e-41d7-bdd9-2c98b3288abe" (UID: "43ec38ca-388e-41d7-bdd9-2c98b3288abe"). InnerVolumeSpecName "kube-api-access-fwrwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:47:54.154776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.154751    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:54.154776 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.154776    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrwc\" (UniqueName: \"kubernetes.io/projected/43ec38ca-388e-41d7-bdd9-2c98b3288abe-kube-api-access-fwrwc\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:54.154950 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.154787    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec38ca-388e-41d7-bdd9-2c98b3288abe-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:54.516115 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:47:54.516079    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:47:54.795522 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.795442    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"
Oct 31 00:47:54.795661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.795445    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm" event={"ID":"43ec38ca-388e-41d7-bdd9-2c98b3288abe","Type":"ContainerDied","Data":"2de9ccab38e5479ee832749d83ded2cc53c34b7c0c58c0dbc32c650f4c00b0cc"}
Oct 31 00:47:54.795661 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:47:54.795599    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de9ccab38e5479ee832749d83ded2cc53c34b7c0c58c0dbc32c650f4c00b0cc"
Oct 31 00:48:00.960442 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:00.960407    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"]
Oct 31 00:48:00.974864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:00.974829    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"]
Oct 31 00:48:01.765194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:01.765152    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"]
Oct 31 00:48:01.768590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:01.768567    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b0db87-z4rdm"]
Oct 31 00:48:02.194117 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:02.194086    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ec38ca-388e-41d7-bdd9-2c98b3288abe" path="/var/lib/kubelet/pods/43ec38ca-388e-41d7-bdd9-2c98b3288abe/volumes"
Oct 31 00:48:04.578003 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:04.577967    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:05.961443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:05.961403    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerName="payload-processing" containerID="cri-o://b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6" gracePeriod=25
Oct 31 00:48:05.975316 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:05.975280    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerName="payload-pre-processing" containerID="cri-o://7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7" gracePeriod=25
Oct 31 00:48:06.386011 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:06.385974    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:48:08.925510 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:08.925451    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:11.376358 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:11.376324    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:11.376826 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:11.376396    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:14.591683 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:14.591613    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:15.437384 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:15.437350    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:48:15.437581 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:15.437351    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:48:23.969338 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:23.969297    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:24.608398 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:24.605800    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:25.436559 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:25.436521    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:48:25.436948 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:25.436601    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:48:26.225275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.225213    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:48:26.264190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.264165    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:48:26.368539 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368513    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2\" (UniqueName: \"kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2\") pod \"7a29cf60-52ea-4968-9a8b-068513cc3553\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") "
Oct 31 00:48:26.368716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368556    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume\") pod \"7a29cf60-52ea-4968-9a8b-068513cc3553\" (UID: \"7a29cf60-52ea-4968-9a8b-068513cc3553\") "
Oct 31 00:48:26.368716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368584    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume\") pod \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") "
Oct 31 00:48:26.368716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368635    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck\" (UniqueName: \"kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck\") pod \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\" (UID: \"1b01f61b-3974-4fe0-9f3e-db7af5dc4401\") "
Oct 31 00:48:26.368920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368898    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume" pod "1b01f61b-3974-4fe0-9f3e-db7af5dc4401" (UID: "1b01f61b-3974-4fe0-9f3e-db7af5dc4401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:26.368920 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368910    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume" pod "7a29cf60-52ea-4968-9a8b-068513cc3553" (UID: "7a29cf60-52ea-4968-9a8b-068513cc3553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:26.369015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368983    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a29cf60-52ea-4968-9a8b-068513cc3553-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:26.369015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.368997    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-plugins-config-volume\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:26.370957 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.370935    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck" pod "1b01f61b-3974-4fe0-9f3e-db7af5dc4401" (UID: "1b01f61b-3974-4fe0-9f3e-db7af5dc4401"). InnerVolumeSpecName "kube-api-access-h4sck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:26.371054 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.370997    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2" pod "7a29cf60-52ea-4968-9a8b-068513cc3553" (UID: "7a29cf60-52ea-4968-9a8b-068513cc3553"). InnerVolumeSpecName "kube-api-access-z6hh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:26.469578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.469553    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hh2\" (UniqueName: \"kubernetes.io/projected/7a29cf60-52ea-4968-9a8b-068513cc3553-kube-api-access-z6hh2\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:26.469578 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:26.469575    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4sck\" (UniqueName: \"kubernetes.io/projected/1b01f61b-3974-4fe0-9f3e-db7af5dc4401-kube-api-access-h4sck\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:27.008995 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.008961    5078 generic.go:356] "Generic (PLEG): container finished" podID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerID="b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6" exitCode=0
Oct 31 00:48:27.009174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.009004    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"
Oct 31 00:48:27.009174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.009042    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" event={"ID":"7a29cf60-52ea-4968-9a8b-068513cc3553","Type":"ContainerDied","Data":"b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6"}
Oct 31 00:48:27.009174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.009095    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5" event={"ID":"7a29cf60-52ea-4968-9a8b-068513cc3553","Type":"ContainerDied","Data":"f8c38af40978be77087945f46a3e42a94c8b543ee0e8fb19b97552eed5845cdd"}
Oct 31 00:48:27.009174 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.009125    5078 scope.go:122] "RemoveContainer" containerID="b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6"
Oct 31 00:48:27.010816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.010793    5078 generic.go:356] "Generic (PLEG): container finished" podID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerID="7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7" exitCode=0
Oct 31 00:48:27.010922 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.010850    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" event={"ID":"1b01f61b-3974-4fe0-9f3e-db7af5dc4401","Type":"ContainerDied","Data":"7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7"}
Oct 31 00:48:27.010922 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.010889    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9" event={"ID":"1b01f61b-3974-4fe0-9f3e-db7af5dc4401","Type":"ContainerDied","Data":"2528b817f228fef741caaf465e3f4a9eb81d2af7d0b864d8c35fd7b48275ccd0"}
Oct 31 00:48:27.011002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.010942    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"
Oct 31 00:48:27.020131 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.020110    5078 scope.go:122] "RemoveContainer" containerID="b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6"
Oct 31 00:48:27.020451 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:27.020429    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6\": container with ID starting with b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6 not found: ID does not exist" containerID="b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6"
Oct 31 00:48:27.020554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.020490    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6"} err="failed to get container status \"b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6\": rpc error: code = NotFound desc = could not find container \"b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6\": container with ID starting with b1d4166b1ddb1a92d509e1648765ec45578d29e5a99140c71458106e9d6695b6 not found: ID does not exist"
Oct 31 00:48:27.020554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.020513    5078 scope.go:122] "RemoveContainer" containerID="7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7"
Oct 31 00:48:27.028947 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.028928    5078 scope.go:122] "RemoveContainer" containerID="7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7"
Oct 31 00:48:27.029204 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:27.029186    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7\": container with ID starting with 7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7 not found: ID does not exist" containerID="7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7"
Oct 31 00:48:27.029264 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.029209    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7"} err="failed to get container status \"7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7\": rpc error: code = NotFound desc = could not find container \"7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7\": container with ID starting with 7be2a39e3fe1aec7f542f1f04eab1a35d538638ead5c491f69149e68bd999ae7 not found: ID does not exist"
Oct 31 00:48:27.035748 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.035729    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"]
Oct 31 00:48:27.040834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.040815    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b0db87-55c468687r24g5"]
Oct 31 00:48:27.053756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.053738    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"]
Oct 31 00:48:27.057801 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:27.057784    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b0db87-7cc96m54g9"]
Oct 31 00:48:28.193287 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:28.193254    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" path="/var/lib/kubelet/pods/1b01f61b-3974-4fe0-9f3e-db7af5dc4401/volumes"
Oct 31 00:48:28.193726 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:28.193679    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" path="/var/lib/kubelet/pods/7a29cf60-52ea-4968-9a8b-068513cc3553/volumes"
Oct 31 00:48:32.308723 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:32.308640    5078 scope.go:122] "RemoveContainer" containerID="8cf285a8e0b69b196a1daa56d622953ee9fdd100aca98d9daff53e7e6d00e2b0"
Oct 31 00:48:34.988495 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:34.988387    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:36.359708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:36.359673    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:48:36.360454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:36.360403    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:36.537795 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:36.536851    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:48:38.928391 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:38.928350    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:45.070832 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:48:45.070797    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec38ca_388e_41d7_bdd9_2c98b3288abe.slice/crio-d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:47.111092 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111058    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"]
Oct 31 00:48:47.111626 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111608    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerName="payload-processing"
Oct 31 00:48:47.111684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111628    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerName="payload-processing"
Oct 31 00:48:47.111684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111645    5078 cpu_manager.go:414] "containerMap: removing container" podUID="43ec38ca-388e-41d7-bdd9-2c98b3288abe" containerName="revoke-keys"
Oct 31 00:48:47.111684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111650    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ec38ca-388e-41d7-bdd9-2c98b3288abe" containerName="revoke-keys"
Oct 31 00:48:47.111684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111668    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerName="payload-pre-processing"
Oct 31 00:48:47.111684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111674    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerName="payload-pre-processing"
Oct 31 00:48:47.111849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111753    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a29cf60-52ea-4968-9a8b-068513cc3553" containerName="payload-processing"
Oct 31 00:48:47.111849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111763    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ec38ca-388e-41d7-bdd9-2c98b3288abe" containerName="revoke-keys"
Oct 31 00:48:47.111849 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.111771    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01f61b-3974-4fe0-9f3e-db7af5dc4401" containerName="payload-pre-processing"
Oct 31 00:48:47.116827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.116810    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.123670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.123627    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:48:47.123670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.123657    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:48:47.123883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.123714    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:48:47.123883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.123827    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:48:47.124823 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.124797    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"]
Oct 31 00:48:47.139509 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.139482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.139690 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.139669    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.139746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.139728    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5m9\" (UniqueName: \"kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.240724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.240684    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.240724 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.240728    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5m9\" (UniqueName: \"kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.241000 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.240777    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.241436 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.241412    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.243424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.243403    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.254044 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.254022    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5m9\" (UniqueName: \"kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9\") pod \"maas-api-revoke-sub-452b6d7b-7lmvw\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.432408 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.432378    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:47.718492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:47.718446    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"]
Oct 31 00:48:47.721915 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:48:47.721883    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03606860_3509_4b88_9005_9c6fb2f424fc.slice/crio-575b3472e0723152d15d2267ea3860d6bbf8420d2943d8160ff63edc99477c78 WatchSource:0}: Error finding container 575b3472e0723152d15d2267ea3860d6bbf8420d2943d8160ff63edc99477c78: Status 404 returned error can't find the container with id 575b3472e0723152d15d2267ea3860d6bbf8420d2943d8160ff63edc99477c78
Oct 31 00:48:48.026688 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:48.026564    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"]
Oct 31 00:48:48.029740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:48.029711    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1f03d5-rtrfh"]
Oct 31 00:48:48.178327 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:48.178283    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw" event={"ID":"03606860-3509-4b88-9005-9c6fb2f424fc","Type":"ContainerStarted","Data":"afbf5343f080d79ab8fac598e8780a1973ed4e91a03edcc7b42b5871f963cc16"}
Oct 31 00:48:48.179066 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:48.179037    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw" event={"ID":"03606860-3509-4b88-9005-9c6fb2f424fc","Type":"ContainerStarted","Data":"575b3472e0723152d15d2267ea3860d6bbf8420d2943d8160ff63edc99477c78"}
Oct 31 00:48:48.195831 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:48.195802    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846fbd26-e8b3-4725-9ccd-58d96d07b035" path="/var/lib/kubelet/pods/846fbd26-e8b3-4725-9ccd-58d96d07b035/volumes"
Oct 31 00:48:48.201837 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:48.201783    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw" podStartSLOduration=1.201764247 podStartE2EDuration="1.201764247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:48.199784637 +0000 UTC m=+3296.146155355" watchObservedRunningTime="2026-10-31 00:48:48.201764247 +0000 UTC m=+3296.148134965"
Oct 31 00:48:49.189387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:49.189346    5078 generic.go:356] "Generic (PLEG): container finished" podID="03606860-3509-4b88-9005-9c6fb2f424fc" containerID="afbf5343f080d79ab8fac598e8780a1973ed4e91a03edcc7b42b5871f963cc16" exitCode=0
Oct 31 00:48:49.189961 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:49.189599    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw" event={"ID":"03606860-3509-4b88-9005-9c6fb2f424fc","Type":"ContainerDied","Data":"afbf5343f080d79ab8fac598e8780a1973ed4e91a03edcc7b42b5871f963cc16"}
Oct 31 00:48:51.379651 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.379563    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:48:51.380213 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.380181    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:48:51.386682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.384677    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw" event={"ID":"03606860-3509-4b88-9005-9c6fb2f424fc","Type":"ContainerDied","Data":"575b3472e0723152d15d2267ea3860d6bbf8420d2943d8160ff63edc99477c78"}
Oct 31 00:48:51.386682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.384760    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575b3472e0723152d15d2267ea3860d6bbf8420d2943d8160ff63edc99477c78"
Oct 31 00:48:51.393653 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.393628    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:51.457124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.457087    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca\") pod \"03606860-3509-4b88-9005-9c6fb2f424fc\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") "
Oct 31 00:48:51.457409 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.457387    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9\" (UniqueName: \"kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9\") pod \"03606860-3509-4b88-9005-9c6fb2f424fc\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") "
Oct 31 00:48:51.457687 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.457665    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca\") pod \"03606860-3509-4b88-9005-9c6fb2f424fc\" (UID: \"03606860-3509-4b88-9005-9c6fb2f424fc\") "
Oct 31 00:48:51.458854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.458829    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca" pod "03606860-3509-4b88-9005-9c6fb2f424fc" (UID: "03606860-3509-4b88-9005-9c6fb2f424fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:51.467076 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.467040    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca" pod "03606860-3509-4b88-9005-9c6fb2f424fc" (UID: "03606860-3509-4b88-9005-9c6fb2f424fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:51.489797 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.489743    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9" pod "03606860-3509-4b88-9005-9c6fb2f424fc" (UID: "03606860-3509-4b88-9005-9c6fb2f424fc"). InnerVolumeSpecName "kube-api-access-qx5m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:51.567489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.559245    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:51.567489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.559279    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03606860-3509-4b88-9005-9c6fb2f424fc-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:51.567489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:51.559295    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5m9\" (UniqueName: \"kubernetes.io/projected/03606860-3509-4b88-9005-9c6fb2f424fc-kube-api-access-qx5m9\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.391428 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:52.391401    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"
Oct 31 00:48:54.616519 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:54.616491    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:48:54.620275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:54.620254    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:48:54.622195 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:54.622175    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:48:54.625795 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:48:54.625779    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:49:01.052102 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.052065    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"]
Oct 31 00:49:01.052800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.052776    5078 cpu_manager.go:414] "containerMap: removing container" podUID="03606860-3509-4b88-9005-9c6fb2f424fc" containerName="revoke-keys"
Oct 31 00:49:01.052901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.052803    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03606860-3509-4b88-9005-9c6fb2f424fc" containerName="revoke-keys"
Oct 31 00:49:01.053065 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.053037    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="03606860-3509-4b88-9005-9c6fb2f424fc" containerName="revoke-keys"
Oct 31 00:49:01.060854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.060806    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.064014 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.063991    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"]
Oct 31 00:49:01.075888 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.075782    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:49:01.080569 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.080545    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:49:01.080807 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.080789    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:49:01.086228 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.081681    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:49:01.218488 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.196195    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.226641 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.226601    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjbq\" (UniqueName: \"kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.241590 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.241560    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.348252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.348154    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.348595 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.348569    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.348792 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.348774    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjbq\" (UniqueName: \"kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.351034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.351007    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.364905 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.364882    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.367619 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.367593    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjbq\" (UniqueName: \"kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq\") pod \"maas-api-revoke-sub-3e31a506-bkf82\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:01.413926 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:01.413899    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:02.644682 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:02.639022    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"]
Oct 31 00:49:02.761611 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:02.761522    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82" event={"ID":"4e32fa14-f227-4dad-9749-899d0cd10ef3","Type":"ContainerStarted","Data":"b33e2e4e83d93b509baa202c4aa9a363c0c9b3a4d89690340f17b7fa06301405"}
Oct 31 00:49:02.968240 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:02.968167    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:49:02.968627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:02.968589    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:49:03.060541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:03.051669    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:49:03.060541 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:03.051752    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:49:03.781833 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:03.781739    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82" event={"ID":"4e32fa14-f227-4dad-9749-899d0cd10ef3","Type":"ContainerStarted","Data":"b73075ea04e2af492b57de898a72e60f0ec432dbd1272f67a022e6f9e7cc7c88"}
Oct 31 00:49:03.803375 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:03.803315    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82" podStartSLOduration=2.803301989 podStartE2EDuration="2.803301989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:03.801263625 +0000 UTC m=+3311.747634354" watchObservedRunningTime="2026-10-31 00:49:03.803301989 +0000 UTC m=+3311.749672770"
Oct 31 00:49:04.792301 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:04.792251    5078 generic.go:356] "Generic (PLEG): container finished" podID="4e32fa14-f227-4dad-9749-899d0cd10ef3" containerID="b73075ea04e2af492b57de898a72e60f0ec432dbd1272f67a022e6f9e7cc7c88" exitCode=0
Oct 31 00:49:04.792869 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:04.792444    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82" event={"ID":"4e32fa14-f227-4dad-9749-899d0cd10ef3","Type":"ContainerDied","Data":"b73075ea04e2af492b57de898a72e60f0ec432dbd1272f67a022e6f9e7cc7c88"}
Oct 31 00:49:06.364584 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:06.364548    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wxssj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:49:06.365194 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:06.365159    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wxssj" podUID="98f5e737-fb2d-4485-9501-c3f7f06146d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:49:06.839174 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:49:06.838809    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:49:06.864714 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:06.864680    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:06.868543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:06.868250    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82" event={"ID":"4e32fa14-f227-4dad-9749-899d0cd10ef3","Type":"ContainerDied","Data":"b33e2e4e83d93b509baa202c4aa9a363c0c9b3a4d89690340f17b7fa06301405"}
Oct 31 00:49:06.868543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:06.868315    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b33e2e4e83d93b509baa202c4aa9a363c0c9b3a4d89690340f17b7fa06301405"
Oct 31 00:49:06.868543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:06.868380    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"
Oct 31 00:49:07.099820 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.099777    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca\") pod \"4e32fa14-f227-4dad-9749-899d0cd10ef3\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") "
Oct 31 00:49:07.114490 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.110275    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq\" (UniqueName: \"kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq\") pod \"4e32fa14-f227-4dad-9749-899d0cd10ef3\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") "
Oct 31 00:49:07.114893 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.114855    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca\") pod \"4e32fa14-f227-4dad-9749-899d0cd10ef3\" (UID: \"4e32fa14-f227-4dad-9749-899d0cd10ef3\") "
Oct 31 00:49:07.141637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.123629    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca" pod "4e32fa14-f227-4dad-9749-899d0cd10ef3" (UID: "4e32fa14-f227-4dad-9749-899d0cd10ef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:07.141637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.139439    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca" pod "4e32fa14-f227-4dad-9749-899d0cd10ef3" (UID: "4e32fa14-f227-4dad-9749-899d0cd10ef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:07.142134 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.142095    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq" pod "4e32fa14-f227-4dad-9749-899d0cd10ef3" (UID: "4e32fa14-f227-4dad-9749-899d0cd10ef3"). InnerVolumeSpecName "kube-api-access-4mjbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:07.142545 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.142507    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:07.142713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.142697    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e32fa14-f227-4dad-9749-899d0cd10ef3-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:07.143058 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:07.143028    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjbq\" (UniqueName: \"kubernetes.io/projected/4e32fa14-f227-4dad-9749-899d0cd10ef3-kube-api-access-4mjbq\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:17.942491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:17.937606    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:49:17.942491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:17.937745    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:49:18.037662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:18.037620    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:49:18.037881 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:18.037725    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:49:19.025523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:19.025484    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"]
Oct 31 00:49:19.028034 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:19.028009    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xbv7"]
Oct 31 00:49:20.193051 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:20.193012    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c500154-30a5-478a-9eff-f94e43655527" path="/var/lib/kubelet/pods/7c500154-30a5-478a-9eff-f94e43655527/volumes"
Oct 31 00:49:21.704236 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.704201    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"]
Oct 31 00:49:21.704868 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.704846    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4e32fa14-f227-4dad-9749-899d0cd10ef3" containerName="revoke-keys"
Oct 31 00:49:21.704964 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.704871    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e32fa14-f227-4dad-9749-899d0cd10ef3" containerName="revoke-keys"
Oct 31 00:49:21.705025 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.705014    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e32fa14-f227-4dad-9749-899d0cd10ef3" containerName="revoke-keys"
Oct 31 00:49:21.709355 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.709333    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.714329 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.714308    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\""
Oct 31 00:49:21.715204 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.715182    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:49:21.715523 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.715501    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:49:21.715814 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.715789    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:49:21.716594 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.716574    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"]
Oct 31 00:49:21.802551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.802519    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.802707 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.802564    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.802756 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.802708    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfkd\" (UniqueName: \"kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.904234 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.904204    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfkd\" (UniqueName: \"kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.904435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.904261    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.904435 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.904291    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.904955 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.904936    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.906900 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.906881    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:21.918141 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:21.918116    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfkd\" (UniqueName: \"kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd\") pod \"maas-api-revoke-sub-d80aeef6-zc55p\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:22.024738 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:22.024684    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:22.299036 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:22.299007    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"]
Oct 31 00:49:22.300098 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:49:22.300069    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b64b1a_e3bf_435e_86db_efc17ff20324.slice/crio-d1aa5cc3f0d44f53bd9cb04660671f0ce228a7f3b80f20ff87deeb423dc9c7b6 WatchSource:0}: Error finding container d1aa5cc3f0d44f53bd9cb04660671f0ce228a7f3b80f20ff87deeb423dc9c7b6: Status 404 returned error can't find the container with id d1aa5cc3f0d44f53bd9cb04660671f0ce228a7f3b80f20ff87deeb423dc9c7b6
Oct 31 00:49:23.107387 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:23.107363    5078 generic.go:356] "Generic (PLEG): container finished" podID="d9b64b1a-e3bf-435e-86db-efc17ff20324" containerID="c008c957a57879358d9a395f3acb0954adbc4a465e698d05c80770df0b549aa9" exitCode=0
Oct 31 00:49:23.107734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:23.107443    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p" event={"ID":"d9b64b1a-e3bf-435e-86db-efc17ff20324","Type":"ContainerDied","Data":"c008c957a57879358d9a395f3acb0954adbc4a465e698d05c80770df0b549aa9"}
Oct 31 00:49:23.107734 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:23.107571    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p" event={"ID":"d9b64b1a-e3bf-435e-86db-efc17ff20324","Type":"ContainerStarted","Data":"d1aa5cc3f0d44f53bd9cb04660671f0ce228a7f3b80f20ff87deeb423dc9c7b6"}
Oct 31 00:49:24.260665 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.260636    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:24.322712 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.322686    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca\") pod \"d9b64b1a-e3bf-435e-86db-efc17ff20324\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") "
Oct 31 00:49:24.322859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.322765    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd\" (UniqueName: \"kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd\") pod \"d9b64b1a-e3bf-435e-86db-efc17ff20324\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") "
Oct 31 00:49:24.322859 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.322804    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca\") pod \"d9b64b1a-e3bf-435e-86db-efc17ff20324\" (UID: \"d9b64b1a-e3bf-435e-86db-efc17ff20324\") "
Oct 31 00:49:24.323215 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.323173    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca" pod "d9b64b1a-e3bf-435e-86db-efc17ff20324" (UID: "d9b64b1a-e3bf-435e-86db-efc17ff20324"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:24.325015 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.324993    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd" pod "d9b64b1a-e3bf-435e-86db-efc17ff20324" (UID: "d9b64b1a-e3bf-435e-86db-efc17ff20324"). InnerVolumeSpecName "kube-api-access-tqfkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:24.325115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.325037    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca" pod "d9b64b1a-e3bf-435e-86db-efc17ff20324" (UID: "d9b64b1a-e3bf-435e-86db-efc17ff20324"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:24.423500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.423446    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:24.423500 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.423492    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfkd\" (UniqueName: \"kubernetes.io/projected/d9b64b1a-e3bf-435e-86db-efc17ff20324-kube-api-access-tqfkd\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:24.423667 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:24.423505    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64b1a-e3bf-435e-86db-efc17ff20324-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:25.128253 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:25.128217    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p" event={"ID":"d9b64b1a-e3bf-435e-86db-efc17ff20324","Type":"ContainerDied","Data":"d1aa5cc3f0d44f53bd9cb04660671f0ce228a7f3b80f20ff87deeb423dc9c7b6"}
Oct 31 00:49:25.128430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:25.128260    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"
Oct 31 00:49:25.128430 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:25.128274    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1aa5cc3f0d44f53bd9cb04660671f0ce228a7f3b80f20ff87deeb423dc9c7b6"
Oct 31 00:49:34.107193 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.107155    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"]
Oct 31 00:49:34.107778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.107729    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d9b64b1a-e3bf-435e-86db-efc17ff20324" containerName="revoke-keys"
Oct 31 00:49:34.107778 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.107745    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b64b1a-e3bf-435e-86db-efc17ff20324" containerName="revoke-keys"
Oct 31 00:49:34.107934 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.107837    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b64b1a-e3bf-435e-86db-efc17ff20324" containerName="revoke-keys"
Oct 31 00:49:34.112371 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.112347    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.118786 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.118760    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:49:34.122901 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.122877    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"]
Oct 31 00:49:34.123706 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.123687    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ldwtt\""
Oct 31 00:49:34.123979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.123960    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 00:49:34.123979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.123960    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 00:49:34.219560 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.219526    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:49:34.219761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.219562    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.219761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.219699    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9582g\" (UniqueName: \"kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.219761 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.219739    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.220012 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.219863    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.321393 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.321351    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.321628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.321477    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9582g\" (UniqueName: \"kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.321628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.321511    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.321628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.321559    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.322170 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.322140    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.324730 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.324708    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.324846 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.324826    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.335074 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.335048    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9582g\" (UniqueName: \"kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g\") pod \"maas-api-6676c7f7bc-zhw8p\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.429313 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.429286    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:34.722496 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.719226    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"]
Oct 31 00:49:34.723790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:34.723758    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:49:35.188580 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:35.188542    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p" event={"ID":"594ec9f1-bc61-46b3-9798-b1373d51bb99","Type":"ContainerStarted","Data":"b41984b0d1a21fb147eaae680f15f661daa5d6106051bf8278f5a9d38a8ef2db"}
Oct 31 00:49:36.195368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:36.195330    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p" event={"ID":"594ec9f1-bc61-46b3-9798-b1373d51bb99","Type":"ContainerStarted","Data":"1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4"}
Oct 31 00:49:36.195791 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:36.195492    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:36.218104 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:36.218064    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p" podStartSLOduration=1.7655064870000001 podStartE2EDuration="2.218053228s" totalImagesPullingTime="452.546741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:36.216695864 +0000 UTC m=+3344.163066595" watchObservedRunningTime="2026-10-31 00:49:36.218053228 +0000 UTC m=+3344.164423937"
Oct 31 00:49:36.386207 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:49:36.386175    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:49:41.691492 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.691434    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"]
Oct 31 00:49:41.692010 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.691699    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p" podUID="594ec9f1-bc61-46b3-9798-b1373d51bb99" containerName="maas-api" containerID="cri-o://1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4" gracePeriod=30
Oct 31 00:49:41.698686 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.698661    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:41.726396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.726372    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"]
Oct 31 00:49:41.730210 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.730191    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.744205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.744180    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"]
Oct 31 00:49:41.791538 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.791503    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a659d5-2e6a-407c-bb87-d2ebadab44ef-maas-api-tls\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.791708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.791555    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6787v\" (UniqueName: \"kubernetes.io/projected/35a659d5-2e6a-407c-bb87-d2ebadab44ef-kube-api-access-6787v\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.791779 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.791705    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a659d5-2e6a-407c-bb87-d2ebadab44ef-metrics-certs\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.791840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.791774    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a659d5-2e6a-407c-bb87-d2ebadab44ef-openshift-service-ca\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.892440 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.892407    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a659d5-2e6a-407c-bb87-d2ebadab44ef-maas-api-tls\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.892629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.892449    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6787v\" (UniqueName: \"kubernetes.io/projected/35a659d5-2e6a-407c-bb87-d2ebadab44ef-kube-api-access-6787v\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.892629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.892513    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a659d5-2e6a-407c-bb87-d2ebadab44ef-metrics-certs\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.892629 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.892545    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a659d5-2e6a-407c-bb87-d2ebadab44ef-openshift-service-ca\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.893282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.893242    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a659d5-2e6a-407c-bb87-d2ebadab44ef-openshift-service-ca\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.895907 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.895874    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a659d5-2e6a-407c-bb87-d2ebadab44ef-metrics-certs\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.896038 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.895945    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a659d5-2e6a-407c-bb87-d2ebadab44ef-maas-api-tls\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.915244 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.915213    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6787v\" (UniqueName: \"kubernetes.io/projected/35a659d5-2e6a-407c-bb87-d2ebadab44ef-kube-api-access-6787v\") pod \"maas-api-8598cb9b7-skrjf\" (UID: \"35a659d5-2e6a-407c-bb87-d2ebadab44ef\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:41.988586 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:41.988560    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:42.089540 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.089512    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:42.094673 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.094649    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs\") pod \"594ec9f1-bc61-46b3-9798-b1373d51bb99\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") "
Oct 31 00:49:42.094816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.094732    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca\") pod \"594ec9f1-bc61-46b3-9798-b1373d51bb99\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") "
Oct 31 00:49:42.094816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.094771    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g\" (UniqueName: \"kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g\") pod \"594ec9f1-bc61-46b3-9798-b1373d51bb99\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") "
Oct 31 00:49:42.094816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.094801    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls\") pod \"594ec9f1-bc61-46b3-9798-b1373d51bb99\" (UID: \"594ec9f1-bc61-46b3-9798-b1373d51bb99\") "
Oct 31 00:49:42.095380 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.095339    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca" pod "594ec9f1-bc61-46b3-9798-b1373d51bb99" (UID: "594ec9f1-bc61-46b3-9798-b1373d51bb99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:42.097181 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.097154    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls" pod "594ec9f1-bc61-46b3-9798-b1373d51bb99" (UID: "594ec9f1-bc61-46b3-9798-b1373d51bb99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:42.097282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.097210    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs" pod "594ec9f1-bc61-46b3-9798-b1373d51bb99" (UID: "594ec9f1-bc61-46b3-9798-b1373d51bb99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:42.097551 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.097526    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g" pod "594ec9f1-bc61-46b3-9798-b1373d51bb99" (UID: "594ec9f1-bc61-46b3-9798-b1373d51bb99"). InnerVolumeSpecName "kube-api-access-9582g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:42.195790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.195760    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-maas-api-tls\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:42.195790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.195785    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594ec9f1-bc61-46b3-9798-b1373d51bb99-metrics-certs\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:42.195790 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.195794    5078 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ec9f1-bc61-46b3-9798-b1373d51bb99-openshift-service-ca\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:42.196106 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.195803    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9582g\" (UniqueName: \"kubernetes.io/projected/594ec9f1-bc61-46b3-9798-b1373d51bb99-kube-api-access-9582g\") on node \"ip-10-0-143-16.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:42.229095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.229030    5078 generic.go:356] "Generic (PLEG): container finished" podID="594ec9f1-bc61-46b3-9798-b1373d51bb99" containerID="1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4" exitCode=0
Oct 31 00:49:42.229095 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.229081    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"
Oct 31 00:49:42.229275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.229112    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p" event={"ID":"594ec9f1-bc61-46b3-9798-b1373d51bb99","Type":"ContainerDied","Data":"1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4"}
Oct 31 00:49:42.229275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.229172    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p" event={"ID":"594ec9f1-bc61-46b3-9798-b1373d51bb99","Type":"ContainerDied","Data":"b41984b0d1a21fb147eaae680f15f661daa5d6106051bf8278f5a9d38a8ef2db"}
Oct 31 00:49:42.229275 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.229196    5078 scope.go:122] "RemoveContainer" containerID="1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4"
Oct 31 00:49:42.238718 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.238700    5078 scope.go:122] "RemoveContainer" containerID="1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4"
Oct 31 00:49:42.239023 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:49:42.238999    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4\": container with ID starting with 1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4 not found: ID does not exist" containerID="1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4"
Oct 31 00:49:42.239072 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.239032    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4"} err="failed to get container status \"1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4\": rpc error: code = NotFound desc = could not find container \"1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4\": container with ID starting with 1a62d102c86dbc17557a11c916008bc4f012b3320c801d51d93bc037c5b303b4 not found: ID does not exist"
Oct 31 00:49:42.250172 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.250151    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"]
Oct 31 00:49:42.253684 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.253665    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-zhw8p"]
Oct 31 00:49:42.372875 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:42.372849    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"]
Oct 31 00:49:42.374536 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:49:42.374504    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a659d5_2e6a_407c_bb87_d2ebadab44ef.slice/crio-8e2ed9cbb825e54a9a8ba805b7fb338c63b22e51dd234ec5366de57fe4e8f7f0 WatchSource:0}: Error finding container 8e2ed9cbb825e54a9a8ba805b7fb338c63b22e51dd234ec5366de57fe4e8f7f0: Status 404 returned error can't find the container with id 8e2ed9cbb825e54a9a8ba805b7fb338c63b22e51dd234ec5366de57fe4e8f7f0
Oct 31 00:49:43.235962 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:43.235926    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf" event={"ID":"35a659d5-2e6a-407c-bb87-d2ebadab44ef","Type":"ContainerStarted","Data":"a0f9242d9f8d63a7e8343a041a6ee54c5e8699c6d515b2ec86a5a13bf0b04286"}
Oct 31 00:49:43.236454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:43.235991    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf" event={"ID":"35a659d5-2e6a-407c-bb87-d2ebadab44ef","Type":"ContainerStarted","Data":"8e2ed9cbb825e54a9a8ba805b7fb338c63b22e51dd234ec5366de57fe4e8f7f0"}
Oct 31 00:49:43.236454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:43.236145    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:49:43.260811 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:43.260761    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf" podStartSLOduration=1.752628728 podStartE2EDuration="2.260747269s" totalImagesPullingTime="508.118541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:43.257283422 +0000 UTC m=+3351.203654141" watchObservedRunningTime="2026-10-31 00:49:43.260747269 +0000 UTC m=+3351.207117986"
Oct 31 00:49:44.198550 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:44.198517    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594ec9f1-bc61-46b3-9798-b1373d51bb99" path="/var/lib/kubelet/pods/594ec9f1-bc61-46b3-9798-b1373d51bb99/volumes"
Oct 31 00:49:49.268489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:49:49.267056    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-skrjf"
Oct 31 00:50:06.386416 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:50:06.386331    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:50:32.356549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:50:32.356516    5078 scope.go:122] "RemoveContainer" containerID="044b11a202aa754ed40db4d385a1530f7a3d9390b7120f579c5a135affe2caaf"
Oct 31 00:50:32.366762 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:50:32.366738    5078 scope.go:122] "RemoveContainer" containerID="8855f164e343caef42bbbc1942525495adc1ce4ba5bd6dd15ade4843e9d0ef75"
Oct 31 00:50:32.375001 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:50:32.374979    5078 scope.go:122] "RemoveContainer" containerID="5fa2690a3ed13f20d6eff4ff56fda3abf83c97a3bc22d09a7edb4b834a12b403"
Oct 31 00:50:36.386402 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:50:36.386365    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:51:06.386879 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:51:06.386835    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:51:19.024471 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:51:19.024434    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"]
Oct 31 00:51:19.030227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:51:19.030202    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-psp8j"]
Oct 31 00:51:20.194220 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:51:20.194185    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551413cd-902f-4a94-a697-0e4ed15b968f" path="/var/lib/kubelet/pods/551413cd-902f-4a94-a697-0e4ed15b968f/volumes"
Oct 31 00:51:36.386355 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:51:36.386277    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:51:40.032294 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:51:40.032257    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"]
Oct 31 00:51:40.034321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:51:40.034301    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rqcsn"]
Oct 31 00:51:40.193930 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:51:40.193896    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6072c2-5488-43fd-b49f-52516d75e4f6" path="/var/lib/kubelet/pods/6f6072c2-5488-43fd-b49f-52516d75e4f6/volumes"
Oct 31 00:52:06.386181 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:52:06.386145    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:52:32.415222 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:32.415192    5078 scope.go:122] "RemoveContainer" containerID="90718bc14604cce6f0105b56af446b18ef24b85093a3917e6755db7d26909361"
Oct 31 00:52:36.386375 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:52:36.386339    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:52:43.022713 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:43.022679    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"]
Oct 31 00:52:43.025029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:43.025001    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6289aae-nnb8l"]
Oct 31 00:52:44.195190 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:44.195154    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f1979f-da2d-42f2-a47e-b901809ab96e" path="/var/lib/kubelet/pods/52f1979f-da2d-42f2-a47e-b901809ab96e/volumes"
Oct 31 00:52:47.946612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:47.946570    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6plnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:52:47.947252 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:47.947218    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6plnx" podUID="eb2f60f1-58ff-4202-a48a-fb057254c182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:52:48.046662 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:48.046620    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:52:48.047016 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:52:48.046980    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-mr9zl" podUID="91ec5a7e-3f44-42e3-907f-c431831b4065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:53:06.385695 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:53:06.385616    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:53:11.117268 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:11.117233    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58544c6c77-l9nc7_fdac0d2e-793d-4fad-a635-528229ed9df1/authorino/0.log"
Oct 31 00:53:22.632693 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:22.632660    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86f7d849b4-vgljn_6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152/manager/1.log"
Oct 31 00:53:23.100167 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:23.100123    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mpl9v_e9bb7055-cf51-4182-a092-2364ed3d5444/manager/1.log"
Oct 31 00:53:24.678067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:24.678036    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86f7d849b4-vgljn_6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152/manager/0.log"
Oct 31 00:53:24.919524 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:24.919456    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mpl9v_e9bb7055-cf51-4182-a092-2364ed3d5444/manager/0.log"
Oct 31 00:53:25.713691 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:25.713664    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8598cb9b7-skrjf_35a659d5-2e6a-407c-bb87-d2ebadab44ef/maas-api/0.log"
Oct 31 00:53:26.096368 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:26.096286    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-bkf82_4e32fa14-f227-4dad-9749-899d0cd10ef3/revoke-keys/0.log"
Oct 31 00:53:26.274717 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:26.274692    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-7lmvw_03606860-3509-4b88-9005-9c6fb2f424fc/revoke-keys/0.log"
Oct 31 00:53:26.661980 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:26.661955    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d80aeef6-zc55p_d9b64b1a-e3bf-435e-86db-efc17ff20324/revoke-keys/0.log"
Oct 31 00:53:28.701953 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:28.701918    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58544c6c77-l9nc7_fdac0d2e-793d-4fad-a635-528229ed9df1/authorino/0.log"
Oct 31 00:53:28.926994 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:28.926958    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ljnt9_bf1eaf8e-40da-42fe-bb27-6900f2488745/manager/1.log"
Oct 31 00:53:29.559834 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:29.559810    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cv2kn_eb0173b8-debd-4739-80ee-070c7445a663/manager/0.log"
Oct 31 00:53:30.477397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:30.477368    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ljnt9_bf1eaf8e-40da-42fe-bb27-6900f2488745/manager/0.log"
Oct 31 00:53:31.130346 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:31.130307    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv_c108abe8-dc34-426e-9a4e-becaf4e882b5/istio-proxy/0.log"
Oct 31 00:53:32.386752 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:32.386720    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6plnx_eb2f60f1-58ff-4202-a48a-fb057254c182/istio-proxy/0.log"
Oct 31 00:53:32.436754 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:32.436724    5078 scope.go:122] "RemoveContainer" containerID="3ce92bf98407cc232bac516500862154fd3e22193ae936232a0dc0a1257d09a2"
Oct 31 00:53:32.592997 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:32.592975    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/1.log"
Oct 31 00:53:33.194772 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:33.194744    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-mr9zl_91ec5a7e-3f44-42e3-907f-c431831b4065/istio-proxy/0.log"
Oct 31 00:53:33.590740 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:33.590672    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/1.log"
Oct 31 00:53:33.988822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:33.988787    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b_922f74f1-f495-4764-b2c9-1f40b2bcf849/istio-proxy/0.log"
Oct 31 00:53:34.196002 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:34.195973    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-c2w2g_fe713694-148e-4e08-bc8a-becc4f002df5/payload-pre-processing/0.log"
Oct 31 00:53:34.583952 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:34.583924    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/1.log"
Oct 31 00:53:35.270118 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:35.270081    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/0.log"
Oct 31 00:53:35.877798 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:35.877765    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/0.log"
Oct 31 00:53:36.280341 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:36.280312    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:53:36.385532 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:53:36.385500    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:53:36.854432 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:36.854397    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk_ee668f0d-fda1-4731-83cf-954fa3780710/main/0.log"
Oct 31 00:53:37.263393 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:37.263364    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-67244_dc6821bd-245e-4dd1-a279-d34358f5d3a1/main/0.log"
Oct 31 00:53:37.675606 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:37.675583    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv_3307b72b-996a-4b66-823c-e4347c25e18f/main/0.log"
Oct 31 00:53:38.277018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:38.276992    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs_7aef86cd-1369-4407-a6a7-4febffe6d133/main/0.log"
Oct 31 00:53:39.294840 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:39.294808    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58544c6c77-l9nc7_fdac0d2e-793d-4fad-a635-528229ed9df1/authorino/0.log"
Oct 31 00:53:49.309217 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:49.309188    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86f7d849b4-vgljn_6cc5fbbb-ba8e-4b51-ba9c-d4efbba92152/manager/1.log"
Oct 31 00:53:49.743303 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:49.743275    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mpl9v_e9bb7055-cf51-4182-a092-2364ed3d5444/manager/1.log"
Oct 31 00:53:51.708424 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:51.708398    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8598cb9b7-skrjf_35a659d5-2e6a-407c-bb87-d2ebadab44ef/maas-api/0.log"
Oct 31 00:53:52.023489 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:52.023389    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"]
Oct 31 00:53:52.025708 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:52.025682    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7lmvw"]
Oct 31 00:53:52.113816 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:52.113788    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-bkf82_4e32fa14-f227-4dad-9749-899d0cd10ef3/revoke-keys/0.log"
Oct 31 00:53:52.193867 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:52.193838    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03606860-3509-4b88-9005-9c6fb2f424fc" path="/var/lib/kubelet/pods/03606860-3509-4b88-9005-9c6fb2f424fc/volumes"
Oct 31 00:53:52.701239 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:52.701210    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d80aeef6-zc55p_d9b64b1a-e3bf-435e-86db-efc17ff20324/revoke-keys/0.log"
Oct 31 00:53:54.333760 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:54.333715    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58544c6c77-l9nc7_fdac0d2e-793d-4fad-a635-528229ed9df1/authorino/0.log"
Oct 31 00:53:54.581975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:54.581948    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ljnt9_bf1eaf8e-40da-42fe-bb27-6900f2488745/manager/1.log"
Oct 31 00:53:54.654115 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:54.654090    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:53:54.658745 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:54.658726    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:53:54.659716 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:54.659698    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:53:54.664627 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:54.664592    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:53:55.220671 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:55.220640    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cv2kn_eb0173b8-debd-4739-80ee-070c7445a663/manager/0.log"
Oct 31 00:53:56.240731 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:56.240700    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv_c108abe8-dc34-426e-9a4e-becaf4e882b5/istio-proxy/0.log"
Oct 31 00:53:57.436979 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:57.436942    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6plnx_eb2f60f1-58ff-4202-a48a-fb057254c182/istio-proxy/0.log"
Oct 31 00:53:57.641073 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:57.641043    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/1.log"
Oct 31 00:53:58.222263 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:58.222229    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-mr9zl_91ec5a7e-3f44-42e3-907f-c431831b4065/istio-proxy/0.log"
Oct 31 00:53:58.618207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:58.618132    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/1.log"
Oct 31 00:53:59.009923 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:59.009898    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b_922f74f1-f495-4764-b2c9-1f40b2bcf849/istio-proxy/0.log"
Oct 31 00:53:59.203800 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:59.203770    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-c2w2g_fe713694-148e-4e08-bc8a-becc4f002df5/payload-pre-processing/0.log"
Oct 31 00:53:59.609421 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:53:59.609389    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/1.log"
Oct 31 00:54:00.174227 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:00.174199    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mp4zk_ee668f0d-fda1-4731-83cf-954fa3780710/main/0.log"
Oct 31 00:54:00.554836 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:00.554753    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-67244_dc6821bd-245e-4dd1-a279-d34358f5d3a1/main/0.log"
Oct 31 00:54:00.934269 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:00.934244    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z9tzv_3307b72b-996a-4b66-823c-e4347c25e18f/main/0.log"
Oct 31 00:54:01.509397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:01.509370    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-q2hgs_7aef86cd-1369-4407-a6a7-4febffe6d133/main/0.log"
Oct 31 00:54:06.386048 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:54:06.386002    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:54:07.023692 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:07.023659    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"]
Oct 31 00:54:07.025609 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:07.025590    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bkf82"]
Oct 31 00:54:08.195071 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:08.195039    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e32fa14-f227-4dad-9749-899d0cd10ef3" path="/var/lib/kubelet/pods/4e32fa14-f227-4dad-9749-899d0cd10ef3/volumes"
Oct 31 00:54:25.024426 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:25.024390    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"]
Oct 31 00:54:25.027321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:25.027302    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80aeef6-zc55p"]
Oct 31 00:54:26.193493 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:26.193439    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b64b1a-e3bf-435e-86db-efc17ff20324" path="/var/lib/kubelet/pods/d9b64b1a-e3bf-435e-86db-efc17ff20324/volumes"
Oct 31 00:54:32.458029 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:32.457945    5078 scope.go:122] "RemoveContainer" containerID="a9db68b4047b2198e553ed146693853a6470cf407f03dd6b7600538194466061"
Oct 31 00:54:32.466925 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:54:32.466906    5078 scope.go:122] "RemoveContainer" containerID="d28341ee6eabadaf7af8d5d12c079846c90193554475ffee30f55db47be2437b"
Oct 31 00:54:36.386685 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:54:36.386638    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:55:06.386289 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:55:06.386246    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:55:32.491161 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:32.491129    5078 scope.go:122] "RemoveContainer" containerID="c008c957a57879358d9a395f3acb0954adbc4a465e698d05c80770df0b549aa9"
Oct 31 00:55:32.499932 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:32.499909    5078 scope.go:122] "RemoveContainer" containerID="b73075ea04e2af492b57de898a72e60f0ec432dbd1272f67a022e6f9e7cc7c88"
Oct 31 00:55:32.510637 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:32.510598    5078 scope.go:122] "RemoveContainer" containerID="afbf5343f080d79ab8fac598e8780a1973ed4e91a03edcc7b42b5871f963cc16"
Oct 31 00:55:36.386840 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:55:36.386800    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"
Oct 31 00:55:40.408456 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:40.408425    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ktssl_e73a9af6-0664-409b-a096-83a990ec98ce/global-pull-secret-syncer/0.log"
Oct 31 00:55:40.470996 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:40.470965    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2qk9k_be5dad9f-114f-4b12-bd51-217374c94fc0/connection-checker/0.log"
Oct 31 00:55:40.503485 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:40.503435    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-69tlv_51bd9499-546b-4b4e-8350-7501c25b0ab2/connection-checker/0.log"
Oct 31 00:55:40.534822 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:40.534805    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kgs9w_25170a0c-24d2-48b8-80fa-d2d2780a41a3/connection-checker/0.log"
Oct 31 00:55:40.718260 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:40.718185    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s8g6n_65fab40a-d3b0-4bf1-9ab3-396dfccebe5f/konnectivity-agent/0.log"
Oct 31 00:55:40.819248 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:40.819225    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-16.us-east-2.compute.internal_fe6922bb420b9fc1aa0231980b4f658c/haproxy/0.log"
Oct 31 00:55:47.055992 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:47.055959    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58544c6c77-l9nc7_fdac0d2e-793d-4fad-a635-528229ed9df1/authorino/0.log"
Oct 31 00:55:47.147568 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:47.147540    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ljnt9_bf1eaf8e-40da-42fe-bb27-6900f2488745/manager/1.log"
Oct 31 00:55:47.150669 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:47.150651    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ljnt9_bf1eaf8e-40da-42fe-bb27-6900f2488745/manager/0.log"
Oct 31 00:55:47.278649 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:47.278625    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cv2kn_eb0173b8-debd-4739-80ee-070c7445a663/manager/0.log"
Oct 31 00:55:49.060053 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:49.060019    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-l5ctp_a91f2be6-7bda-48a0-a0fc-8d5da8469dd7/cluster-monitoring-operator/0.log"
Oct 31 00:55:49.263397 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:49.263365    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kpj57_c25f317f-5d84-42c6-b06d-e69dbb04420e/node-exporter/0.log"
Oct 31 00:55:49.292291 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:49.292261    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kpj57_c25f317f-5d84-42c6-b06d-e69dbb04420e/kube-rbac-proxy/0.log"
Oct 31 00:55:49.324417 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:49.324364    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kpj57_c25f317f-5d84-42c6-b06d-e69dbb04420e/init-textfile/0.log"
Oct 31 00:55:50.005554 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.005520    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-94hrt_d46ade2d-3c27-496c-b4c4-d0a80f55f9a0/prometheus-operator-admission-webhook/1.log"
Oct 31 00:55:50.005735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.005609    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-94hrt_d46ade2d-3c27-496c-b4c4-d0a80f55f9a0/prometheus-operator-admission-webhook/0.log"
Oct 31 00:55:50.341119 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.341050    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"]
Oct 31 00:55:50.341494 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.341487    5078 cpu_manager.go:414] "containerMap: removing container" podUID="594ec9f1-bc61-46b3-9798-b1373d51bb99" containerName="maas-api"
Oct 31 00:55:50.341543 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.341500    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594ec9f1-bc61-46b3-9798-b1373d51bb99" containerName="maas-api"
Oct 31 00:55:50.341612 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.341602    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="594ec9f1-bc61-46b3-9798-b1373d51bb99" containerName="maas-api"
Oct 31 00:55:50.346321 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.346300    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.351453 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.351431    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bbcn2\"/\"default-dockercfg-qbh96\""
Oct 31 00:55:50.353864 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.353839    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"]
Oct 31 00:55:50.354850 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.354825    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbcn2\"/\"kube-root-ca.crt\""
Oct 31 00:55:50.356975 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.356956    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbcn2\"/\"openshift-service-ca.crt\""
Oct 31 00:55:50.440150 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.440122    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-sys\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.440324 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.440193    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhx9\" (UniqueName: \"kubernetes.io/projected/502d1881-4569-4a6a-97ac-07f0f5485973-kube-api-access-9xhx9\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.440828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.440399    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-podres\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.440828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.440444    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-proc\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.440828 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.440510    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-lib-modules\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541342 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541314    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhx9\" (UniqueName: \"kubernetes.io/projected/502d1881-4569-4a6a-97ac-07f0f5485973-kube-api-access-9xhx9\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541387    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-podres\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541415    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-proc\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541440    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-lib-modules\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541549 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541544    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-proc\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-podres\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541595    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-sys\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541767 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541647    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-lib-modules\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.541870 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.541766    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502d1881-4569-4a6a-97ac-07f0f5485973-sys\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.554670 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.554646    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhx9\" (UniqueName: \"kubernetes.io/projected/502d1881-4569-4a6a-97ac-07f0f5485973-kube-api-access-9xhx9\") pod \"perf-node-gather-daemonset-9qf8l\" (UID: \"502d1881-4569-4a6a-97ac-07f0f5485973\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.669709 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.669677    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:50.949290 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:50.948930    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"]
Oct 31 00:55:50.949451 ip-10-0-143-16 kubenswrapper[5078]: W1031 00:55:50.949356    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod502d1881_4569_4a6a_97ac_07f0f5485973.slice/crio-eeccf52b2ecca8417079a46919419347c80e35ef513bd607ea69b9294a92597e WatchSource:0}: Error finding container eeccf52b2ecca8417079a46919419347c80e35ef513bd607ea69b9294a92597e: Status 404 returned error can't find the container with id eeccf52b2ecca8417079a46919419347c80e35ef513bd607ea69b9294a92597e
Oct 31 00:55:51.497781 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:51.497747    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l" event={"ID":"502d1881-4569-4a6a-97ac-07f0f5485973","Type":"ContainerStarted","Data":"0b88ba05ac4f30ef4b3a1ac8290091e230bade772eeae4db432ac3a923eb34f2"}
Oct 31 00:55:51.498192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:51.497806    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l" event={"ID":"502d1881-4569-4a6a-97ac-07f0f5485973","Type":"ContainerStarted","Data":"eeccf52b2ecca8417079a46919419347c80e35ef513bd607ea69b9294a92597e"}
Oct 31 00:55:51.498192 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:51.497844    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:51.520799 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:51.520755    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l" podStartSLOduration=1.520742024 podStartE2EDuration="1.520742024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:51.520228298 +0000 UTC m=+3719.466599015" watchObservedRunningTime="2026-10-31 00:55:51.520742024 +0000 UTC m=+3719.467112760"
Oct 31 00:55:51.596817 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:51.596797    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-k7rhq_1b949609-5560-4b48-a5b7-229d7430d7c0/networking-console-plugin/0.log"
Oct 31 00:55:52.401622 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:52.401582    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/3.log"
Oct 31 00:55:52.406197 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:52.406170    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zzxg7_578a9d2b-c161-4a91-8f5a-8cf5a891a2bb/console-operator/2.log"
Oct 31 00:55:53.003062 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:53.003036    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6wdkk_b6209fdf-df1c-4a4f-b5ce-55812e9f07a4/download-server/1.log"
Oct 31 00:55:53.003593 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:53.003434    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6wdkk_b6209fdf-df1c-4a4f-b5ce-55812e9f07a4/download-server/0.log"
Oct 31 00:55:53.633605 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:53.633575    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-sf4kv_b25a3f44-1814-43a8-a01d-f98c870b7d9f/volume-data-source-validator/0.log"
Oct 31 00:55:54.992330 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:54.992289    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-shskm_d1384f8a-a2ad-4e55-b6e5-14ab134fa489/dns/0.log"
Oct 31 00:55:55.023311 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:55.023284    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-shskm_d1384f8a-a2ad-4e55-b6e5-14ab134fa489/kube-rbac-proxy/0.log"
Oct 31 00:55:55.058928 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:55.058906    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ccfzb_3bbf2354-7b35-4374-adc3-e694b71f6e9e/dns-node-resolver/0.log"
Oct 31 00:55:55.811827 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:55.811799    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c895f9d65-tkw4d_2f6a88c2-1453-4033-8027-afc8b69d3ee7/registry/0.log"
Oct 31 00:55:55.878391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:55.878362    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bndnt_6e24b7db-f2a1-4500-9d82-963292cb7765/node-ca/0.log"
Oct 31 00:55:57.124473 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:57.124426    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv_c108abe8-dc34-426e-9a4e-becaf4e882b5/istio-proxy/0.log"
Oct 31 00:55:57.514081 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:57.514057    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-9qf8l"
Oct 31 00:55:58.028529 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.028496    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6plnx_eb2f60f1-58ff-4202-a48a-fb057254c182/istio-proxy/0.log"
Oct 31 00:55:58.065031 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.065003    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/0.log"
Oct 31 00:55:58.065171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.065088    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/1.log"
Oct 31 00:55:58.165366 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.165343    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-mr9zl_91ec5a7e-3f44-42e3-907f-c431831b4065/istio-proxy/0.log"
Oct 31 00:55:58.249915 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.249891    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/1.log"
Oct 31 00:55:58.250124 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.250107    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/0.log"
Oct 31 00:55:58.326571 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.326507    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b_922f74f1-f495-4764-b2c9-1f40b2bcf849/istio-proxy/0.log"
Oct 31 00:55:58.367391 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.367369    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-c2w2g_fe713694-148e-4e08-bc8a-becc4f002df5/payload-pre-processing/0.log"
Oct 31 00:55:58.456589 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.456568    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/1.log"
Oct 31 00:55:58.456854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:55:58.456829    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:56:00.919454 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:00.919425    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv_c108abe8-dc34-426e-9a4e-becaf4e882b5/istio-proxy/0.log"
Oct 31 00:56:00.992657 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:00.992628    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrwcz_7eb92403-d07f-4bc4-bf34-4a15adff4863/serve-healthcheck-canary/0.log"
Oct 31 00:56:01.214491 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.214386    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6plnx_eb2f60f1-58ff-4202-a48a-fb057254c182/istio-proxy/0.log"
Oct 31 00:56:01.236033 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.236001    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/0.log"
Oct 31 00:56:01.236205 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.236042    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/1.log"
Oct 31 00:56:01.296050 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.296021    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-mr9zl_91ec5a7e-3f44-42e3-907f-c431831b4065/istio-proxy/0.log"
Oct 31 00:56:01.337445 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.337379    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/1.log"
Oct 31 00:56:01.337625 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.337593    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/0.log"
Oct 31 00:56:01.383018 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.382986    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b_922f74f1-f495-4764-b2c9-1f40b2bcf849/istio-proxy/0.log"
Oct 31 00:56:01.402602 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.402572    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-c2w2g_fe713694-148e-4e08-bc8a-becc4f002df5/payload-pre-processing/0.log"
Oct 31 00:56:01.450883 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.450851    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:56:01.451067 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.450923    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/1.log"
Oct 31 00:56:01.670600 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.670572    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-mb6hw_c4cac373-8436-4b03-a28b-0fa1bf8e7d13/insights-operator/0.log"
Oct 31 00:56:01.672207 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.672182    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-mb6hw_c4cac373-8436-4b03-a28b-0fa1bf8e7d13/insights-operator/1.log"
Oct 31 00:56:01.705917 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.705894    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgtpq_567d58b9-e17a-41fd-a5e2-6ec9ff24597d/kube-rbac-proxy/0.log"
Oct 31 00:56:01.738636 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.738601    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgtpq_567d58b9-e17a-41fd-a5e2-6ec9ff24597d/exporter/0.log"
Oct 31 00:56:01.772728 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:01.772700    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgtpq_567d58b9-e17a-41fd-a5e2-6ec9ff24597d/extractor/0.log"
Oct 31 00:56:02.063171 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:02.063090    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-gt7cj-g4r94_967c5ac1-949a-4f15-94d6-a4fa8342ee73/insights-gathering/0.log"
Oct 31 00:56:04.538443 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.538411    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67b5c995fcf8qvv_c108abe8-dc34-426e-9a4e-becaf4e882b5/istio-proxy/0.log"
Oct 31 00:56:04.807176 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.807100    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6plnx_eb2f60f1-58ff-4202-a48a-fb057254c182/istio-proxy/0.log"
Oct 31 00:56:04.825282 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.825255    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-58f644b8bd-tm95b_654d9cdc-9ff9-4d6d-96e7-35fe4f90894f/manager/0.log"
Oct 31 00:56:04.827628 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.827609    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/1.log"
Oct 31 00:56:04.827758 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.827673    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-7kqrm_e1fbc0c4-9520-446a-826b-20866760c19a/istio-proxy/0.log"
Oct 31 00:56:04.856084 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.856061    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-58f644b8bd-tm95b_654d9cdc-9ff9-4d6d-96e7-35fe4f90894f/copy-manifests/0.log"
Oct 31 00:56:04.878080 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.878057    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-mr9zl_91ec5a7e-3f44-42e3-907f-c431831b4065/istio-proxy/0.log"
Oct 31 00:56:04.917911 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.917886    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/1.log"
Oct 31 00:56:04.918117 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.918100    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-wxssj_98f5e737-fb2d-4485-9501-c3f7f06146d4/istio-proxy/0.log"
Oct 31 00:56:04.962746 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.962720    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-vrw5b_922f74f1-f495-4764-b2c9-1f40b2bcf849/istio-proxy/0.log"
Oct 31 00:56:04.981854 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:04.981836    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-c2w2g_fe713694-148e-4e08-bc8a-becc4f002df5/payload-pre-processing/0.log"
Oct 31 00:56:05.022396 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:05.022370    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/0.log"
Oct 31 00:56:05.022735 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:05.022724    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74cc75bf99-fkb87_d7e8e526-6b9d-4aff-9bd3-3b38c1461aa1/router/1.log"
Oct 31 00:56:06.111407 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:06.111378    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-657fcc8b8f-z9psz_3c0d9ceb-7c2b-4872-a065-4bfe4b332064/manager/1.log"
Oct 31 00:56:06.111877 ip-10-0-143-16 kubenswrapper[5078]: I1031 00:56:06.111598    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-657fcc8b8f-z9psz_3c0d9ceb-7c2b-4872-a065-4bfe4b332064/manager/0.log"
Oct 31 00:56:06.386973 ip-10-0-143-16 kubenswrapper[5078]: E1031 00:56:06.386939    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgtpq" containerName="extractor"